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Preface 


This  paper  was  prepared  for  a  Department  of  Defense  (DoD)  project  directed  by  the  Office 
of  Program  Analysis  and  Evaluation  (PA&E)  and  sponsored  by  the  Defense  Modeling  and 
Simulation  Office  (now  the  DoD  Modeling  and  Simulation  Coordination  Office  [M&S  CO]). 
That  project  is  supporting  development  of  DoD’s  Analysis  Modeling  and  Simulation  Master 
Plan  (AMSMP),  which  will  reflect  needs  for  modeling  and  simulation  (M&S)*  of  the  “analysis 
domain,”  i.e.,  the  domain  of  defense-planning  functions,  such  as  capability  assessments  and 
program  analysis.  Separate  master  plans  are  being  prepared  for  the  acquisition,  experimenta¬ 
tion,  training,  (operational)  planning,  and  testing  domains. 

The  Institute  for  Defense  Analyses  (IDA)  has  led  the  effort  with  participation  by  the 
RAND  and  MITRE  Corporations.  The  project  has  involved  white  papers,  workshops  inform¬ 
ing  the  iteration  of  those  white  papers,  and  a  draft  master  plan.  The  present  paper  (one  of  the 
white  papers)  relates  to  DoD-level  analysis,  analysis  practices,  and  implications  for  M&S,  or  what 
may  be  referred  to  simply  as  methods.  Other  white  papers  address  tools — i.e.,  M&S  for  tradi¬ 
tional  warfare,  M&S  for  nontraditional  warfare,  data,  and  technologies  for  M&S7 

As  discussed  in  Appendixes  A  and  B,  this  paper  benefited  from  stimulating  working- 
group  deliberations  in  two  workshops,  held  March  14-16,  2006,  and  June  26-27,  2006.  The 
former  was  comprised  largely  of  invited  representatives  from  government  and  federally  funded 
research  and  development  centers;  the  latter  highlighted  suggestions  from  invited  industry  rep¬ 
resentatives.  Although  we  benefited  from  these  interactions,  we  are  solely  responsible  for  this 
paper’s  final  content.  Comments  are  welcome  and  should  be  addressed  to  the  senior  author, 
Paul  K.  Davis  (pdavis@rand.org)  at  the  RAND  Corporation.  Amy  Henninger  is  with  the 
Institute  for  Defense  Analyses. 

The  RAND  portion  of  the  research  presented  here  was  conducted  within  the  Acquisi¬ 
tion  and  Technology  Policy  Center  of  the  RAND  National  Defense  Research  Institute,  a 
federally  funded  research  and  development  center  sponsored  by  the  Office  of  the  Secretary  of 
Defense,  the  Joint  Staff,  the  Unified  Combatant  Commands,  the  Department  of  the  Navy, 
the  Marine  Corps,  the  defense  agencies,  and  the  defense  Intelligence  Community.  For  more 
information  on  the  center,  contact  the  Director,  Philip  Anton,  who  can  be  reached  by  email  at 


In  this  paper,  we  use  M&S  to  mean  modeling  and  simulation  or  models  and  simulations,  depending  on  the  context. 

!  Portions  of  the  research  reported  here  were  developed  in  connection  with  a  National  Academy  study  (National  Research 
Council,  2006)  and  a  study  on  military  decisionmaking  (Davis  and  Kahan,  forthcoming).  As  a  result,  some  of  the  material 
is  similar  in  those  publications  and  the  present  paper. 
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atpc-director@rand.org),  by  phone  at  310-393-0411,  extension  7798;  or  by  mail  at  the  RAND 
Corporation,  1776  Main  St.,  Santa  Monica,  California  90407-2138.  More  information  about 
RAND  is  available  at  www.rand.org. 
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Summary 


The  Challenges 

This  paper  proceeds  from  an  analysis-centric  perspective,  asking  what  should  be  emphasized 
in  M&S  so  as  to  help  provide  future  policymakers  with  insightful  analysis  informing  invest¬ 
ment  choices  within  and  across  joint  capability  areas.  Many  of  DoD’s  analytic  needs  are  long¬ 
standing,  as  is  work  to  develop  and  improve  supporting  M&S.  Some  needs,  however,  are  newer 
and  present  special  challenges  particularly  relevant  to  the  complexities  of  modern  military 
operations.  Those  special  challenges  are  summarized  in  Table  S.l.  They  include  increases  of 
scope,  notably  addressing  what  DoD  calls  traditional,  irregular,  disruptive,  and  catastrophic 
conflicts;  the  proliferation  of  nuclear  states;  and  DoD’s  role  in  homeland  defense.  They  include 
new  planning  paradigms,  notably  capabilities-based  planning,  effects-based  operations,  and 
network-centric  operations.  They  also  include  technology-sensitive  considerations,  such  as 
developing  systems  of  systems  (SoS)  and  capabilities  for  information  operations  (IO).  Although 
we  touch  upon  issues,  primarily  in  connection  with  networking,  we  do  not  do  them  justice. 
They  are,  however,  treated  elsewhere  as  problems  for  the  acquisition  and  operations  commu¬ 
nities.  The  first  six  challenges,  those  of  scope  and  planning  paradigms,  are  the  focus  of  this 
paper. 

Taken  as  a  whole,  the  table  reflects  the  fact  that  warfare  is  moving  beyond  the  relatively 
well  understood  phenomena  that  can  often  be  treated  by  “physics  models”  into  the  less  under¬ 
stood  area  of  human  behavior  and  how  to  affect  it. 

The  following  are  the  special  challenges  for  DoD  analysis  (the  challenges  highlighted 
appear  to  us  to  be  the  most  demanding  of  attention,  although  many  others  exist): 

Scope 

•  full-spectrum  planning:  traditional,  irregular,  disruptive,  and  catastrophic  conflicts 

•  strategy  for  a  world  with  more  proliferated  nuclear  weapons 

•  DoD  support  for  homeland  defense. 

Planning  Paradigms 

•  capabilities-based  planning  (CBP)  for  dealing  with  profound  uncertainty 

•  effects-based  operations  (EBO) 

•  network-centric  operations. 
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Technology-Driven  Considerations 

•  systems  of  systems  (SoS) 

•  information  operations  (IO,  which  is  not  treated  further  in  this  paper). 


Functional  Requirements 

Given  these  challenges  for  analysis,  what  are  the  implications  for  M&S?  Our  conclusion  is  that 
the  priority  should  be  on  meeting  the  following  functional  requirements  for  M&S: 

1 .  Routine  and  perceptive  treatment  of  uncertainty,  including  deep  uncertainty. 

2.  Emphasizing  flexible,  adaptive,  and  robust  strategies  (FAR  strategies). 

3.  Adaptive  models  able  to  evaluate  candidate  FAR  strategies  (i.e.,  models  that  represent 
adaptation  of  commanders  to  circumstances). 

4.  Adopting  a  family-of-tools  approach  that  includes  human-in-the-loop  tools. 

This  list  has  a  logically  deductive  flow.  Because  reliable  prediction  is  often  simply  not  in  the 
cards  (after  all,  our  analysis  must  deal  with  complex  adaptive  systems),  planning  under  uncer¬ 
tainty  implies  an  emphasis  on  FAR  strategies.  To  assess  such  strategies,  we  must  use  exploratory 
analysis  supported  by  a  special  class  of  M&S  suitable  for  varying  a  myriad  of  assumptions.* 
Moreover,  we  cannot  expect  fully  computerized  M&S  to  do  the  job  because  they  are  simply 
not  yet  competitive  in  many  cases  with  human  war  games  and  expert  judgment.  This  is  so 
particularly  in  efforts  to  understand  potential  effects  in  “PMESII  space” — i.e.,  the  space  of 
interacting  political,  military,  economic,  social,  infrastructure,  and  information  factors.  The 
limitations  of  traditional  M&S  lead  us  to  a  family-of  tools  approach  that  includes  a  prominent 
role  for  humans.  The  approach  contrasts  markedly  with  associating  analysis  with  constructive, 
rather  than  virtual  or  live,  simulation. 

Table  S.l  shows  some  of  the  many  possible  tools  or  instruments  that  can  be  employed, 
each  of  which  has  different  strengths  and  weaknesses.  The  evaluations,  shown  as  cell  shadings 
in  the  table,  depend  on  many  assumptions;  counterexamples  can  be  found  for  all  of  them. 
Nonetheless,  the  basic  story  conveyed  is  correct  in  the  aggregate.  Starting  with  the  first  row, 
relatively  simple  models  (e.g.,  many  spreadsheet  models  or  their  equivalent  in  other  high-level 
languages)  can  be  easily  understood  and  explained;  they  may  be  exercised  easily  to  quickly  vary 
a  broad  range  of  assumptions  in  a  systematic  exploratory  analysis.  That  is,  they  are  agile.  They 
are  also  “personal” — i.e.,  they  can  be  used  and  adapted  directly  by  the  relevant  analysts.  The 
value  of  relatively  simple  models  is  as  great  today  as  it  was  at  the  advent  of  operations  research, 


Exploratory  analysis  (Davis,  1994;  2003b)  examines  the  behavior  of  a  model  or  ensemble  of  models  across  the  full  space 
of  input-parameter  values.  It  differs  fundamentally  from  ordinary  sensitivity  analysis,  which  usually  assumes  a  base  case 
and  then  varies  inputs  one  at  a  time.  Exploratory  analysis  assumes  no  base  case  (except  perhaps  for  comparison  purposes) 
and  varies  all  parameters  simultaneously.  This  is  most  practical  with  low-resolution  models  with  relatively  few  parameters. 
Exploratory  analysis  may  be  parametric  or  probabilistic. 
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Table  S.1 

Relative  Merits  of  Illustrative  Items  in  a  Family  of  Tools 


Examples 


START,  CAPE,  EXHALT, 
many  unnamed 

THUNDER,  ITEM, 

JICM,  JWARS 

Enhanced  versions  of 
above 

Models  built  with 
Isaac  or  MANA 

Models  built  with 
Repast,  RAND-SEAS, 
GA-MANA 

Janus,  JCATS,  NSS 


NOTES:  The  shading  depends  on  many  implicit  assumptions.  The  key  point  is  that  instrument  classes 
have  different  virtues. 


ABMs:  agent-based  models. 

MRM:  multiresolution  modeling. 
GA-MANA:  a  model  built  with  MANA 
using  genetic  algorithms. 


Very  poor 


Very  good 


RAND  OP176-T-S.1 


systems  analysis,  and  policy  analysis  a  half  century  or  more  ago.  Such  models,  however,  have 
limited  scope  and  do  not  represent  either  physical  or  human  phenomenology  except  in  the 
most  abstract  terms.  In  contrast,  looking  at  the  last  four  rows,  detailed  models  (e.g.,  entity-level 
simulations),  human  war  gaming  (including  seminar  games,  larger  and  more  formal  games, 
perhaps  supported  by  simulations  in  the  background,  and  human-in-the-loop  work  with  simu¬ 
lations),  historical  analysis,  and  held  experiments  have  complementary  characteristics:  They  are 
hardly  agile  and  are  often  not  easy  to  understand  fully,  but  they  can  provide  rich  descriptions 
of  both  physical  and  human  phenomena,  including  those  associated  with  complex  adaptive 
systems  (albeit  with  substantial  attendant  uncertainties). 

Big  strategic  simulations  (in  the  second  row  of  the  table)  are  heavily  used  because  they 
have  substantial  breadth  and  moderate  agility.  In  this  context,  they  usually  have  no  human  in 
the  loop.  Their  breadth  makes  them  very  suitable  for  integrative  joint  work  involving  multiple 
services  and  allies  in  theater  or  multitheater  contexts,  as  in  total-force  planning  or  analysis  of 
capabilities  for  particular  defense-planning  scenarios.  They  describe  physical  events  only  in  very 
aggregate  terms,  however;  and  they  do  not  currently  represent  human  phenomenology  well, 
which  limits  their  applicability  for  studying  the  complexities  of,  say,  counterinsurgency  and 
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stabilization  operations,  or  of  various  efforts  to  deter,  dissuade,  or  coerce  adversaries.  Finally, 
they  are  excessively  complex  for  studying  particular  mission-level  or  capability- area  issues. 

Significantly,  such  strategic  simulations  can  in  principle  be  a  notch  more  powerful,  by 
adding  to  them  adaptive  submodels  representing  commanders  and  group  behaviors,  and  by 
using  them  not  to  examine  particular  scenarios  in  detail  as  is  customary  today,  but  to  conduct 
exploratory  analysis  within  scenario  classes  as  well  as  within  a  given  class.  The  mechanisms 
for  adaptation  can  include  “agents”  as  submodels,  but  also  game-theoretic  algorithms,  control 
theory,  or  various  operations-research  methods.  Were  the  strategic  simulations  more  modular, 
individual  modules  could  be  used  in  studies  of  particular  missions  or  capability  areas. 

Continuing,  “simple  ABMs”  refer  to  simple  agent-based  models,  which  have  become 
quite  significant  in  the  past  decade.  These  models  are  rather  agile  and  have  moderate  abil¬ 
ity  to  represent  human  phenomenology  and  inform  issues  of  command  and  control,  but  they 
typically  have  narrow  scope  and  are  otherwise  quite  limited — in  part  because  they  have  been 
developed  by  people  studying  how  even  simple  low-level  behavioral  rules  can  lead  to  higher- 
level  emergent  phenomena.  That  emphasis  has  been  accompanied  by  a  suppression  of  many 
issues  considered  critical  to  higher-level  military  analysis,  such  as  top-down  direction.  More 
sophisticated  agent-based  models  (the  next  row  of  Table  S.l)  can  include  a  mix  of  top-down 
and  bottom-up  features,  better  physical  simulation,  and  algorithms  for  making  “rational”  deci¬ 
sions  and  explaining  those  decisions,  and  these  can  then  be  included  in  high-quality  physical 
simulations.  Good  abstractions  of  such  sophisticated  ABMs  would  be  ideal  candidates  for  the 
adaptive  submodels  postulated  above  for  advanced  strategic  simulations. 

Although  lip  service  is  often  paid  to  the  family-of-tools  concept,  and  resources  are  some¬ 
times  used  to  inform  and  calibrate  relatively  aggregate  models  from  more  detailed  models,  ana¬ 
lytical  organizations  often  tilt  toward  investment  in  big  constructive  simulations.  We  conclude 
that  DoD  and  service  analytical  organizations  should  adjust  their  implicit  investment  portfolios  to 
greatly  increase  the  efforts  to  make  use  of  simplified  or  specialized  models  suitable  for  exploratory 
analysis,  human  gaming  and  other  use  of  experts,  and  empirical  research  drawing  on  history,  train¬ 
ing,  and  real-world  operations  planning 


Priorities  for  Investment  and  Action 

The  text  of  the  paper  includes  many  relatively  detailed  suggestions  on  how  to  improve  treat¬ 
ment  of  uncertainty,  use  human  gaming  and  experts,  and  apply  agent-based  modeling.  It  sug¬ 
gests  methods  for  better  integrating  and  consolidating  some  of  the  big  simulations  currently 
used  in  DoD’s  strategic  planning.  The  reader  is  urged  to  look  at  these  suggestions,  as  well  as 
this  short  summary.  Table  S.2,  however,  provides  condensed  recommendations  and  priorities 
for  investment,  along  with  rough  estimates  of  costs.  Our  rationale  for  the  priorities  is  that  the 
items  chosen  represent  DoD  public-good  investments.  These  public  goods  include,  e.g.,  efforts 
to  synthesize  disparate  strands  of  ongoing  research  and  provide  primers  that  make  sense  of 
inherently  complex  new  challenges.  Table  S.2  recommends  actions  in  three  classes,  all of which 
are  critical.  Within  each  class,  we  reluctantly  indicate  some  relative  priorities,  but  they  depend 
on  dubious  assumptions.  Final  investment  decisions  should  be  made  in  a  context  of  alternative 
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Table  S.2 

Summary  Recommendations  for  DoD-Level  Investment 


Action  Sponsor/Rough  Cost 

Class  Policy  Action  Investment  Estimate 
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Analysis  should 
emphasize  evalua¬ 
tion  of  alternative 
strategies  for 
potential  flexibility, 
adaptiveness,  and 
robustness 


Increase  openness,  competition,  agility,  and  PA&E/$3-$4M  over 

capability,  including  for  broad,  low-resolution  two  years,  but  with  a 
exploratory  analysis,  by  (1)  integrating  and  net  savings  over  time 

improving  existing  models  to  achieve  a  single, 
integrated,  and  very  modular  multiresolution 
campaign  model;  and  (2)  spinning  off  or 
developing  separately  small,  specialized,  and 
readily  modified  models.  Establish  cross¬ 
calibration  with  service  models.  This  consti¬ 
tutes  a  new  "business  model" 


1  Generalize  and  expand  current  exploratory 
analysis  methods  and  tools  for  portfolio 
work  so  that  related  choices  can  be  robust  to 
uncertainty  about  underlying  assumptions 
about  scenarios,  category  weights,  risks,  costs, 
etc. 


M&S  CO,  AT&L, 
PA&E/$1M  over  two 
years  initially.  Later, 
$2M  for  creating 
products 
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Require  study 
components  such 
as  gaming,  red 
teaming,  use  of 
experts,  and  use 
of  model  families 
representing  all 
PM  ESI  I  dimensions 


Develop  consolidated  primer  on  use  of  gaming, 
red  teaming,  and  use  of  experts  for  higher- 
level  DoD  analysis,  and  for  connecting  results 
to  M&S.  This  should  include  in-depth  discussion 


M&S  CO/$1M-$2M 
for  a  cooperative 
effort  of  two 
organizations  over 
two  years 


Issue  new  VV&A 
guidelines  for 
models  intended 
for  PMESII  work 


3  Commission  report  drawing  upon  community 
knowledge  to  recommend  new  VV&A 
guidelines  focused  more  on  exploration  than 
prediction 
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Require  that  models  1 
permit  optional 
human  play  or 
insertion  of  results 
of  PM  ESI  l-rich 
games 

2 


2 


Require  serious  2 

treatment  of  future 
network-centric 
capabilities  and 
implications 


Develop  review  paper  covering  adaptive  M&S  CO/$500K 

modeling  methods  relevant  to  higher-level  DoD 

work.  This  should  cover  agent-based  models, 

control  theory,  game  theoretic,  and  other 

operations  research  methods 


Experiments  on  PMESII-sensitive  model-and- 
game  families,  aspiring  to  well-established 
relationships  among  low-,  middle-,  and  high- 
resolution  models 


M&S  CO/$1M-$2M 
over  two  years, 
covering  at  least  two 
efforts 


Develop  primer  on  effective  use  of  human-in-  M&S  CO/$1M-$2M 
the-loop  analysis  with  simulations  of  varied  for  a  cooperative 

resolution  and  character  effort  of  two 

organizations  over 
two  years 


Commission  competitive  design  for  new  or 
substantially  reprogrammed  campaign  and 
mission-level  models  built  around  network 
centricity  (and  degrees  thereof).  Priority  2  only 
because  of  high  cost 


Joint  Staff/$2M  for 
industry  competition 
and  review.  $5M- 
$40M  eventual  cost, 
assuming  reuse 


NOTES:  The  number  column  indicates  the  relative  priorities  within  each  category.  AT&L  =  Acquisition, 
Technology,  and  Logistics. 
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budget  levels  over  time  and  an  understanding  of  what  can  and  will  be  funded  through  other 
channels.  To  put  it  differently,  Table  S.2  omits  many  high-priority  investment  items — e.g., 
in  advanced  agent  methods  to  assist  in  modeling  and  analysis  of  culture-sensitive  issues,  only 
because  they  are  apparently  being  pursued  by  the  services,  commands,  and  defense  agencies. 
The  Office  of  the  Secretary  of  Defense  should  ensure  that  this  is  true  and,  if  it  is  not,  arrange 
for  changed  priorities.  Much  of  the  work  needed  is  in  the  province  of  science  and  technology 
and,  thus,  of  the  Director  for  Defense  Research  and  Engineering. 

Within  the  first  action  class  (treatment  of  uncertainty),  the  first  recommendation  is  com¬ 
plex,  calling  for  improvement  of  big  models  and  support  of  simpler  models  (often  specialty 
models)  and  assuring  the  ability  to  compare  the  models.  This  recommendation,  taken  as  a  whole, 
corresponds  to  a  distinctly  new  business  model  for  DoD-level  investment.  As  a  whole,  the  need 
exists  for  greater  openness,  competition,  agility,  and  capabilities.  Past  investment  has  tended 
to  emphasize  standardization  and  the  predictable  result  has  been  to  generate  models  and  mod¬ 
eling  activities  that  are  good  at  what  they  do,  with  consistent  data,  but  that  lack  the  vigor, 
creativity,  and  cross-cutting  flexibility  that  is  needed.  DoD  should  on  the  one  hand  integrate 
and  improve  its  current  strategic  models  (potentially  having  only  a  single  high-level  simulation 
for  core  work,  with  broad  coverage  and  either  links  or  relationships  to  more  detailed  models 
as  needed  for  particular  analyses).  DoD  should  make  the  result  as  modular  as  possible  and 
actively  plan  continuing  competition  and  evolution  at  the  module  level.  Different  module  ver¬ 
sions  will  likely  be  superior  in  different  analyses,  whether  because  of  resolution  or  what  phe¬ 
nomena  need  to  be  represented  well.  Some  module  versions  might  allow  for  optional  human 
play  or  might  be  tuned  to  recent  empirical  information.  DoD  should,  on  the  other  hand, 
also  encourage  and  use  more  relatively  small  “specialty”  models,  whether  for  agile  aggregate- 
level  work  or  for  very  narrow  analysis  of  mission  capabilities — analysis  unencumbered  by  the 
weight  of  full  campaign  models.  Investments  to  develop  tools  for  and  skill  in  rapid  develop¬ 
ment  of  such  specialty  models  will  also  be  needed.  The  tools  could  range  from  those  akin  to 
MATLAB01  to  those  more  like  the  tools  of  the  Defense  Advanced  Research  Projects  Agency’s 
(DARPA’s)  RealWorld  program.  DoD  should  also  invest  modestly  to  ensure  that  researchers 
are  able  readily  to  compare  assumptions  and  results  of,  e.g.,  the  specialty  models  and  relevant 
big-model  modules.  In  some  cases,  a  specialty  model  may  be  identical,  except  for  programming 
language,  to  a  big-model  module;  in  other  cases,  there  may  be  important  and  desirable,  but 
optional,  differences.  Similarly,  cross-comparison  mechanisms  should  exist  to  ensure  that  DoD’s 
integrated  campaign  model  has  understood  relationships  with  relevant  service-level  models. 

This  recommendation  is  complex,  but  we  urge  honoring  its  complexity  rather  than  invest¬ 
ing  only,  for  example,  in  the  consolidation  and  integration  of  current  strategic  models  for  the 
sake  of  greater  efficiency  and  standardization. 

The  second  recommendation  is  to  generalize  the  concepts  of  “exploratory  analysis”  for 
effective  application  in  portfolio  analysis — e.g.,  analysis  to  inform  resource  allocation  within  a 
capability  area  or  across  multiple  capability  areas.  The  allure  of  portfolio  methods  is  consider¬ 
able,  but  the  results  are  often  bogus  because  they  depend  too  sensitively  on  low-level  assump¬ 
tions.  As  with  capabilities  analysis  generally,  the  solution  is  to  seek  flexible,  adaptive,  and 
robust  strategies  of  investment,  which  requires  analysis  in  which  the  assumptions  affecting  the 
resource-allocation  decisions  are  systematically  and  simultaneously  varied. 
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Within  the  second  action  class  (routine  treatment  of  complex  human  and  social  issues), 
our  primary  recommendation  is  for  DoD  to  develop  an  integrative  primer  on  the  use  of  meth¬ 
ods  such  as  red  teaming,  war  gaming,  and  use  of  experts.  This  class  of  issues  is  ripe  for  coherent 
synthesis,  which  is  not  an  effort  for  a  committee.  We  also  mention  the  need  to  revise  policies  on 
verification,  validation,  and  accreditation  (VV&A)  so  that  they  acknowledge  and  deal  sensibly 
with  the  vast  array  of  models  (and  associated  data)  that  is  simply  not  “predictive”  in  the  classic 
sense,  but  rather  is  intended  for  more  exploratory  work  amid  uncertainty.  This  work  can  build 
on  the  substantial  base  of  VV&A  material  developed  by  M&S  CO  over  the  past  decade. 

Within  the  third  action  class  (adaptiveness  and  networking),  we  again  recommend  a  syn¬ 
thetic  activity,  starting  with  one  that  pulls  together  methods  such  as  agent-based  modeling, 
control  theory,  and  game  theory  so  as  to  clarify  which  methods  for  providing  adaptiveness  are 
most  suitable.  We  also  recommend  recognizing  that  a  new  generation  of  strategic/operational- 
level  model  is  needed  for  the  network- centric  era.  Even  JWARS  (now  called  JAS)2  has  only 
some  of  the  features  needed,  and  no  consensus  yet  exists  about  what  a  new-generation  model 
might  look  like.  This  suggests  the  need  for  preliminary  research  and  a  competition  of  ideas. 
This  is  distinct  from  and  in  addition  to  network-centric  models  necessary  for  design  and  evalu¬ 
ation  of  particular  systems  of  systems,  which  are  of  great  importance  within  the  acquisition 
community. 
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ABM 

AT&L 

CAA 

DIME 

DoD 

EA 

EBO 

FAR 

FFRDC 

IDA 

IO 

JFCOM 

M&S 

M&S  CO 

MS&A 

MRM 

MSG 

NCW 

OSD 


agent-based  model 

Acquisition,  Technology,  and  Logistics 
Concepts  Analysis  Agency 

diplomatic,  information,  military,  and  economic 

Department  of  Defense 

exploratory  analysis 

effects-based  operations 

flexible,  adaptive,  and  robust 

federally  funded  research  and  development  center 

Institute  for  Defense  Analyses 

information  operations 

Joint  Forces  Command 

modeling  and  simulation  or  models  and  simulations,  depending  on  the 
context 

DoD  Modeling  and  Simulation  Coordination  Office  (formerly  the  Defense 
Modeling  and  Simulation  Office) 

modeling,  simulation,  and  analysis 

multiresolution  modeling  (allowing  inputs  to  be  made  at  different  levels  of 
resolution) 

massive  scenario  generation 
network  centric  warfare 
Office  of  the  Secretary  of  Defense 
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PA&E 

PMESII 

SoS 

VV&A 


Program  Analysis  and  Evaluation 

political,  military,  economic,  social,  infrastructure,  information 
systems  of  systems 

verification,  validation,  and  accreditation 


CHAPTER  ONE 


Introduction 


Purpose  and  Scope 

This  paper  addresses  analysis  and  analysis  practices  for  defense  planning  and  their  implica¬ 
tions  for  modeling  and  simulation  (M&S).*  The  analysis  in  question  is  accomplished  for  Qua¬ 
drennial  Reviews  and  for  continuing  work  on  capability  assessments,  requirements  analysis, 
and  program  analysis.  The  paper’s  purpose  is  to  delineate  priorities  for  the  way  ahead — i.e., 
for  investments  and  other  actions  to  ensure  that  future  M&S  will  serve  the  needs  of  defense¬ 
planning  analysis. 

Traditionally,  analysis  for  defense  planning  has  been  at  the  strategic  and  operational 
levels,  but  what  these  levels  now  include  is  unclear,  given  that  the  organization  of  forces  is 
changing,  low-level  activities  can  have  strategic  importance,  and  sound  assessment  of  capa¬ 
bilities  sometimes  requires  in-depth  analysis.  Accordingly,  the  paper  takes  a  broad  view  of  its 
scope,  although  not  adequately  addressing  some  issues  (notably  related  to  systems  of  systems 
[SoS],  personnel,  training,  and  operations  planning)  that  are  usually  seen  as  subjects  for  other 
communities.  Future  work  should  address  the  substantial  overlaps  that  increasingly  exist  across 
the  M&S-using  communities. 


Perspective  and  Approach 

Consistent  with  its  purpose,  the  paper  approaches  issues  from  strongly  analysis-centric  and 
capabilities-based  perspectives,  rather  than  the  perspective  of  modeling  per  se}  Chapter  Two 
characterizes  the  new  analytic  problems  implied  by  policy  concerns.  Chapter  Three  draws 
implications  for  M&S,  noting  both  shortfalls  and  ways  of  addressing  them  in  the  near  to  mid 
term.  Chapter  Four  discusses  some  of  the  improvement  methods  in  more  detail  and  suggests 
priorities  for  action  and  investment.  Chapter  Five  draws  some  conclusions.  The  bibliography 
includes  citations  to  relevant  published  literature. 

Overall,  the  paper  is  ambitious,  taking  the  view  that  much  of  what  should  be  done  to 
improve  analysis  can  be  done  sooner  rather  than  later.  Although  science  and  technology  will 
continue  to  provide  grist  for  further  progress,  the  key  concepts,  methods,  and  tools  for  what  is 
needed  most  already  exist  or  are  well  within  reach.  This  said,  organizational  inertia  is  a  significant 


In  this  paper,  we  use  M&S  to  mean  modeling  and  simulation  or  models  and  simulations,  depending  on  the  context. 
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factor  and  assimilation  of  new  methods  and  tools  is  always  challenging.  In  a  sense,  the  issue  is 
transformation  of  the  analytic  community. 


CHAPTER  TWO 


Policymaker  Concerns 


Special  Challenges 

Many  defense-planning  analysis  issues  are  long-standing  and  continuing.  This  paper  makes 
no  attempt  to  review  them  and  assumes  that  evolutionary  progress  will  continue  to  be  made 
in  improving  such  analysis  and  the  M&S  that  supports  it.  Our  focus  is  exclusively  on  “special 
challenges”  and,  in  particular,  how  analysis  can  address  major  Department  of  Defense  (DoD) 
themes  that  have  emerged  over  the  past  decade,  and  that  have  been  emphasized  in  recent  high- 
level  documents  (Rumsfeld,  2006).2  Significantly,  it  is  no  longer  necessary  to  stress  jointness 
because  the  requirement  is  now  widely  accepted. 

The  following  are  the  themes  of  modern  DoD  planning: 

Scope 

•  full-spectrum  planning:  traditional,  irregular,  disruptive,  and  catastrophic  conflicts 

•  operations  abroad  and  DoD  support  for  homeland  defense. 

Planning  Paradigms 

•  capabilities-based  planning  for  dealing  with  profound  uncertainty 

•  effects-based  operations  (EBO) 

•  strategy  for  a  world  with  more  proliferated  nuclear  weapons. 

Technology-Driven  Considerations 

•  systems  of  systems  (SoS) 

•  information  operations/network-centric  operations. 


Observations  About  the  Special  Challenges  and  Themes 

The  following  observations  cut  across  the  themes  shown  above: 

1 .  Complex,  Interconnected  Systems.  It  has  become  crucial  to  understand  links  among  the 
many  factors  at  play;  to  apply  all  of  the  instruments  of  national  power  in  a  coordi¬ 
nated  and  sensible  manner;  and  to  do  so  anticipating  that  the  many  parties  involved 
in  conflict  learn,  adapt,  and  do  unanticipated  things  and  that  other  unanticipated 
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events  will  also  occur.'  All  of  this  is  frequently  discussed  as  planning  for  effects-based 
operations.3  How  best  to  conceive  and  then  achieve  desired  effects  is  a  considerable 
challenge  for  analysis.  It  is  clear,  however,  that  a  key  to  meeting  the  challenge  will 
be  recognizing  that  the  systems  of  interest  are  complex  adaptive  systems  (CAS),  and 
that  many  of  the  concepts  from  modern  research  on  such  systems  will  need  to  be 
applied.4 

2.  Capabilities-Based  Planning.  To  cope  with  uncertainty  and  other  challenges,  DoD 
has  shifted  to  capabilities-based  planning  (Rumsfeld,  2001  and  2006),  which  stresses 
development  and  honing  of  capabilities  that  can  be  applied  to  a  wide  range  of  chal¬ 
lenges  and  circumstances  (Davis,  2002a),  and  to  its  cousin  in  operations — “Adaptive 
Planning,”  in  which  war  plans  are  developed  so  as  to  anticipate  and  then  support 
major  changes  as  circumstances  change.  Implementation  of  capabilities-based  plan¬ 
ning  is  a  difficult  undertaking — in  part  because  the  analytical  community  has  been 
so  wedded  to  in-depth  analysis  rather  than  analysis  under  uncertainty.  Although 
force  transformation  is  sometimes  treated  as  a  separate  issue,  many  of  DoD’s  efforts  at 
transformation  (see  http://www.oft.osd.mil/,  as  of  December  19,  2006)  relate  closely 
to  both  capabilities-based  planning  and  the  next  item,  networking. 

3.  Networking  The  ascendancy  of  ubiquitous  networking  in  all  DoD  activities  is  often 
referred  to  as  moving  to  network- centric  operations.  The  advent  of  networking  also 
increases  the  scope  and  importance  of  information  operations  (IO)  and  revolution¬ 
izes  many  aspects  of  command  and  control.  Unfortunately,  it  remains  difficult  to 
represent  network- centric  issues  appropriately  in  analysis.  Although  it  is  easy  enough 
to  represent  discrete  platforms  and  communication  links,  it  is  not  so  easy  as  yet 
to  reflect  such  core  considerations  as  publish  and  subscribe,  self-synchronization,  or 
the  capability  to  modify  and  adapt  command,  control,  communications,  computers, 
information,  surveillance,  and  reconnaissance  (C4ISR)  networks  substantially  in  the 
course  of  operations,  sometimes  linking  entities  that  had  not  been  expected  to  be 
operating  together.5  The  same  difficulty  arises  in  designing  and  evaluating  systems 
of  systems,  a  key  challenge  for  acquisition.  System  of  systems  work  is  also  crucial  in 
much  of  the  social-networking  analysis  relevant  to  DIME/PMESII  issues. 

4.  Nontraditional  Warfare.  Because  U.S.  military  prowess  in  traditional  combat  has 
become  overwhelming,  adversaries  have  been  avoiding  direct  engagements,  instead 
resorting  to  “irregular”  warfare;  moving  installations  underground  or  otherwise  seek¬ 
ing  to  frustrate  overhead  surveillance  and  precision  targeting;  mingling  with  the  civil 
population;  or  attacking  the  United  States,  its  allies,  and  their  interests  at  home.  The 
global  war  on  terrorism  is  but  one  face  of  this  challenge. 

5.  Stabilization  and  Reconstruction.  The  U.S.  military  has  not  traditionally  been  enthu¬ 
siastic  about  stability  and  reconstruction  missions,  but — as  highlighted  in  the  recent 
Quadrennial  Defense  Review — they  may  well  be  a  major  element  of  military  opera- 


In  DoD’s  acronym-loving  fashion,  it  refers  to  political,  military,  economic,  social,  infrastructure,  and  information 
(PMESII)  factors  and  diplomatic,  information,  military,  and  economic  (DIME)  instruments. 
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tions  over  the  foreseeable  future.  Success  in  these  missions,  without  the  benefit  of 
overwhelming  presence  of  forces,  poses  daunting  challenges. 

6.  Homeland  Security.  Although  we  do  not  discuss  the  issue  further  in  this  paper,  home¬ 
land  security  M&S  challenges  have  become  very  important. 

The  significance  of  these  observations  can  perhaps  best  be  appreciated  by  contrasting  the  types 
of  questions  posed  to  analysts  in  the  previous  era  and  today.  Appendix  C  provides  such  a 
contrast. 


Implications  for  Analysis,  Modeling,  and  Simulation 

It  follows  from  the  demands  described  above  that  a  number  of  key  functional  needs  can  be  iden¬ 
tified  for  analysis  and  supporting  M&S.  These  include: 

1.  Routine  and  Perceptive  Treatment  of  Uncertainty,  Including  Deep  Uncertainty.  Uncer¬ 
tainties  are  profound  regarding  future  conflicts,  the  circumstances  of  those  conflicts, 
and  the  complex  effects  of  actions  taken  within  the  conflicts.  Further,  major  changes 
will  occur  over  time,  affecting  national  priorities  and  defense  programs.  Thus,  analy¬ 
sis  and  M&S  must  deal  well  with  uncertainty  and,  as  part  of  that,  must  characterize 
risks  well. 

2.  Emphasizing  Flexible,  Adaptive,  and  Robust  Strategies  (FAR  strategies).  Because  plan¬ 
ning  and  operations  are  so  heavily  affected  by  uncertainty,  with  accurate  prediction 
often  being  implausible,  strategies  need  to  be  flexible,  adaptive,  and  robust  (FAR 
strategies).  Candidate  strategies  must  therefore  be  evaluated  across  a  broad  possibility 
space.6 

3.  Adaptive  Models  and  Rapid  Model  Development.  Evaluation  of  adaptive  strategies 
requires  adaptive  models.  That  is,  if  strategies  are  conceived  with  branches  and  other 
features  to  cope  with  uncertainty,  their  evaluation  will  require  models  that  gener¬ 
ate  realistically  dynamic  circumstances  with  which  strategies  are  to  deal,  and  that 
represent  the  strategies’  intended  adaptations.  Such  models  will  need  to  reflect  the 
learning,  adapting,  and  sometimes  random  acting  of  individual  groups,  and  also  the 
possibility  of  structural  changes  in  the  nature  of  the  system  as  coalitions  form  or  dis¬ 
solve,  key  leaders  emerge  or  disappear,  and  physical  events  change  realities  of,  say, 
geography  or  access.7  Since  not  all  the  adaptations  can  be  anticipated  in  advance,  it 
will  also  be  important  to  make  rapid  changes  in  models  and  to  develop  quickly  small 
specialized  models  or  submodels. 

4.  Reinserting  People  in  M&S  and  Related  Analysis.  For  dealing  with  uncertainty,  intro¬ 
ducing  innovative  concepts,  or  working  across  boundaries  such  as  those  associated 
with  DIME  and  PMESII,  people  are  exceedingly  capable — much  more  so  than  tra¬ 
ditional  M&S.  Because  of  this,  gaming  is  often  a  preferred  method  for  operational 
planners  and  strategic  planners.8  The  same  human  advantages  are  needed  in  M&S, 
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in  which  optional  human-in-the-loop  simulation  can  be  quite  useful.  Reinserting 
people  can  be  accomplished  as  discussed  in  a  later  chapter. 

These  needs  are  not  merely  nice  to  have;  rather,  they  lie  at  the  core  of  whether  decisionmakers 
are  to  be  well  served. 


CHAPTER  THREE 


Dealing  with  Uncertainty  and  Risk 


Introduction 

The  most  important  step  in  dealing  well  with  uncertainty  is  acknowledging  its  many  dimen¬ 
sions  and  their  magnitude.  Doing  so  is  a  core  element  of  capabilities-based  planning.  In  recent 
years,  the  Department  of  Defense  has  come  a  long  way,  moving  from  an  era  in  which  only 
one  or  two  scenarios  were  taken  seriously  to  one  in  which  planning  guidance  describes  in 
some  detail  quite  a  number  of  substantially  different  name-level  challenge  scenarios.  Many  of 
these  are  examined  with  considerable  care  by  the  Office  of  the  Secretary  of  Defense  (Program 
Analysis  and  Evaluation)  (OSD  [PA&E]),  J-8,  and  the  services’  analytical  shops  in  support  of 
force  planning.  The  next  step  is  for  analysis  to  routinely  and  systematically  examine  capabili¬ 
ties  for  each  such  name-level  scenario  across  the  range  of  possible  circumstances  (including 
creative  strategies  and  tactics  by  both  sides).  That  is,  exploratory  analysis  is  needed  for  each 
name-level  scenario  (Davis,  1994,  2002a,  and  2003b),  as  well  as  across  them.*  To  illustrate  the 
need,  consider  how  the  operations  in  Afghanistan  and  Iraq  have  stressed  U.S.  force  structure 
in  ways  very  different  from  those  assumed  in  force-planning  scenarios  used  in  earlier  years. 
Before  the  invasion  of  Iraq,  operations  planners  had  to  consider  many  possible  plan  variants, 
and  what  developed  still  had  unexpected  features.9  Good  capabilities  analysis,  then,  requires 
broad  exploration. 


Strategic  Adaptive  Planning 

Overview 

One  aspect  of  planning  for  adaptiveness  is  seeking  capabilities  that  will  prove  valuable  for 
many  missions  and  circumstances.  Another  aspect  is  strategic  adaptive  planning,  by  which  we 
mean  pursuing  capability-development  programs  that  are  expected  to  change  over  time  as 
information  is  gained  about  the  future  security  environment,  what  can  actually  be  achieved 


Exploratory  analysis  (Davis,  1994;  2003b)  examines  a  model’s  behavior  across  the  full  space  of  input-parameter  values. 
It  differs  fundamentally  from  ordinary  sensitivity  analysis,  which  usually  assumes  a  base  case  and  then  varies  inputs  one  at 
a  time.  Exploratory  analysis  (EA)  assumes  no  base  case  (except  perhaps  for  comparison  purposes)  and  varies  all  parameters 
simultaneously.  This  is  most  practical  with  low- resolution  models  with  relatively  few  parameters.  EA  may  be  parametric  or 
probabilistic. 
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technologically,  how  alternative  approaches  to  achieving  the  capabilities  compare,  and  the 
preferences  of  political  leadership.  In  this  approach,  a  capabilities-development  program  may 
begin  with  competitive  ideas  carried  forward  through  more  detailed  concept  formulation  and 
even  experimentation  (perhaps  with  prototypes).  Later,  the  program  may  largely  focus  on  a 
particular  path,  but  it  may  be  hedged  by  carrying  along  alternatives  at  a  lower  rate  of  expendi¬ 
ture.  If  appropriately  funded  and  executed,  evolutionary  acquisition  is  an  example  of  strategic 
adaptive  planning. 

An  Example  in  Industry:  Real-Options  Theory 

One  tangible  example  of  deliberate  strategic  adaptive  planning  can  be  seen  in  “real-option 
planning”  in  industry.  To  be  sure,  there  are  stark  differences  between  industrial  planning  and 
defense  planning.  The  most  obvious  is  that  companies  are  focused  primarily  on  a  single  mea¬ 
sure — future  profit — whereas  the  DoD  has  many  objectives  to  pursue,  which  tend  not  to  be 
readily  commensurate,  much  less  in  something  as  simple  as  dollars.  Further,  some  industries 
have  the  great  advantage  of  being  able  to  base  risk- assessment  analysis  on  rich  historical  and 
developmental  databases,  such  as  those  of  the  financial  markets  over  the  past  century,  or  those 
of  aircraft  designers  over  the  past  50  years.  Some  of  DoD’s  risks  are  different  in  character  than 
those  found  in  industry.  So,  yes,  there  are  big  differences. 

Nonetheless,  the  practice  of  real-options  theory  is  a  good  example  of  strategic  adaptive 
planning.  Its  success  in  some  large  companies  is  an  existence  proof  that  such  planning  can 
be  done,  that  chief  executives  are  both  interested  in  and  willing  to  invest  on  the  basis  of  risk- 
and-opportunity  balancing  considerations,  that  such  investments  can  be  informed  by  “hard” 
analysis,  and  that  the  payoffs  can  be  very  large.10 

In  business,  the  real-options  issues  may  include,  e.g.,  deciding  whether  to  invest  in  vari¬ 
ants  of  a  product’s  principal  design  or  in  alternative  products.  If  these  are  assessed  with  only 
best-estimate  assumptions  about  the  future,  they  may  have  little  apparent  value,  but  if  assessed 
with  recognition  of  uncertainty  about  future  demand,  success  of  the  principal  design  (and  the 
options),  and  various  possible  competitors,  such  options  may  have  substantial  value.  The  net 
effect  is  that  some  investments  create  or  maintain  options  that  hedge  against  bad  futures  and / 
or  prepare  to  exploit  good  opportunities. 

Parallels  exist  in  the  defense  context.  The  Missile  Defense  Agency  (MDA),  for  example, 
is  seeking  to  develop  ballistic  missile  defense  systems  that  may  be  used  for  very  different  mis¬ 
sions  (e.g.,  homeland  defense  versus  defense  of  an  ally’s  capital  or  of  forward-deployed  U.S. 
forces)  against  adversary  systems  that  may  have  very  different  countermeasure  characteristics. 
Further,  the  objectives  for  ballistic  missile  defense  systems  in  the  future  may  change  drastically 
as  a  function  of  political  leadership,  the  nature  of  the  strategic  environment,  and  what  proves 
feasible. 

In  such  a  context  of  uncertainty,  MDA’s  program  arguably  needs  to  be  adaptive  enough 
so  that  the  requisite  technology  is  pursued  to  deal  with  as  much  of  the  possibility  space  as  pos¬ 
sible,  given  constraints  of  budget  and  technology.  In  practice,  this  implies  investing  in  a  range 
of  options — but  also  with  the  expectation  of  assessing  progress  on  these  options  over  time  and 
then  choosing — i.e.,  killing  off  some  that  fail  to  develop  successfully  so  as  to  pay  for  those  that 
are  more  successful  and  that  appear  increasingly  appropriate  with  time. 
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Examples  in  Social-Policy  Studies 

Another  approach  to  strategic  adaptive  planning  is  represented  by  work  on  policy  responses 
to  perceived  trends  of  global  warning,  and  in  studies  of  very  long-term  social  issues  (Lempert, 
Popper,  Bankes,  2003).  The  authors  have  variously  referred  to  their  approach  as  exploratory 
modeling  and  robust  adaptive  planning.  The  philosophy  of  the  work  has  much  in  common 
with  real-options  planning,  but  the  mathematical  and  analytical  methods  are  quite  different. 


Obstacles 

At  least  three  obstacles  exist  to  having  exploratory  analysis  in  various  forms  available  routinely 
as  part  of  defense  planning:  (1)  the  current  analytical  infrastructure  and  its  familiar  routines, 
(2)  antipathy  to  uncertainty  analysis,  and  (3)  the  paucity  of  analytical  models  and  methods 
needed  for  exploratory  analysis.  A  few  words  on  each  of  these  are  warranted. 

1.  Familiar  Routines.  In  the  DoD,  falling  into  familiar  routines  translates  into  placing 
great  emphasis  on  conducting  in-depth  analysis  on  defense-planning  scenarios,  with 
models  that  the  services  accept,  and  with  databases  approved  by  the  services  or  other 
authorities.  Achieving  this  is  laudable  as  a  baseline  and,  indeed,  is  a  considerable 
achievement  since  it  promotes  integration  and  facilitates  apples-to-apples  compari¬ 
son  of  study  claims.  Unfortunately,  current  execution  leaves  little  time  for  more  far- 
reaching  investigation  away  from  the  comfortable  approved  baselines.  Policy  guid¬ 
ance  urging  broader  parametric  exploration  and  providing  guidelines  for  doing  so  is 
seldom  followed.11 

2.  Antipathy  to  Uncertainty  Analysis.  Many  analysts  describe  having  been  admonished  by 
their  superiors  that  decisionmakers  want  best  estimates  presented  without  the  compli¬ 
cation  of  caveats.  Such  admonishments  clearly  occur  at  intermediate  levels  between 
working  analysts  and  decisionmakers,  and,  sometimes,  decisionmakers  themselves 
send  such  messages.  However,  analysis  that  suppresses  uncertainty  is  bad  technically 
and  ethically,  as  can  be  appreciated  by  reviewing  classic  texts  and  professional-society 
codes  of  practice.  Furthermore,  the  claim  about  decisionmakers  is  a  canard,  because 
many  top  decisionmakers  welcome  discussions  of  uncertainty  so  long  as  the  discus¬ 
sions  also  assist  in  choice.  Decisionmakers  have  little  time  for  what  they  may  see  as 
hand-wringing,  paralysis,  technically  elegant  but  off-point  excursions,  or  long  lists  of 
caveats.  In  contrast,  when  uncertainty-rich  analysis  assists  in  defining  flexible,  adap¬ 
tive,  and  robust  strategies — whether  by  “stacking  decks”  or  planning  branches  and 
hedges — the  reception  will  often  be  positive.  This  is  true  both  of  decisionmakers  who 
approach  problems  “analytically”  and  of  decisionmakers  who  are  more  intuitively 
oriented  and  predisposed  toward  particular  directions,  but  nonetheless  are  interested 
in  assuring  success,  which  includes  preparing  for  contingencies  (Davis,  Kulick,  and 
Egner,  2005;  Davis  and  Kahan,  2006). 

This  said,  there  are  distinct  limits  to  how  much  uncertainty  analysis  is  useful  and 
in  what  form  it  should  be  presented.  It  has  been  argued  (Davis,  Kulick,  and  Egner, 
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2005)  that  a  suitable  level  of  detail  for  bottom-line  results  is  as  shown  in  Table  3.1 
(although,  of  course,  it  depends  on  further  layers  of  reasoning  that  are  not  shown). 

3.  Paucity  of  Models  and  Methods.  The  third  obstacle  has  been  one  of  models  and  meth¬ 
ods.  This  obstacle  is  discussed  in  the  following  section. 


Tools  and  Methods  for  Exploratory  Analysis 

A  common  lament  by  analytic  organizations  is  that  the  models  available  to  them  are  not  suit¬ 
able  for  far-reaching  exploratory  analysis;  with  effort,  they  can  be  used  to  do  numerous  excur¬ 
sions  around  baseline  cases,  but  going  beyond  that  is  painful,  time-consuming,  and  expensive. 
As  discussed  earlier  in  the  paper,  simpler,  faster-running  models  are  desirable  for  agility  and 
breadth  (i.e.,  for  exploratory  analysis).  The  question  then  arises,  where  can  an  organization  find 
such  models  and  establish  their  validity  for  the  intended  purposes?  Four  primary  options  exist, 
as  indicated  in  Table  3.2. 

The  first  of  these  options  is  attractive  if  one  has  a  model  designed  with  multiple  levels  of 
resolution  in  mind.  In  these  cases,  one  can  turn  off  higher  levels  of  detail  for  particular  mod¬ 
ules  when  doing  exploration  and  then  turn  them  back  on  as  needed  to  follow  up  on  conclu¬ 
sions  in  more  detail.  Such  features  have  been  built  into  only  some  of  the  commonly  available 
DoD  models.  Adding  some  such  features  to  preexisting  models  is  straightforward  in  some 
cases.  In  other  cases,  however,  it  is  more  difficult  and  requires  approximations  and  narrowing 
of  scope. 

Whereas  exploratory  analysis  as  discussed  above  is  usually  performed  with  a  single  model, 
albeit  one  that  is  highly  parameterized  and  perhaps  has  probability  distributions  to  represent 
uncertainty,  this  makes  sense  only  if  the  model  itself  is  sufficiently  sound.  Often,  we  lack  con¬ 
fidence  about  the  model  itself,  referring  to  “structural  uncertainty,”  ambiguity  about  causal 
relationships,  and  other  complications.  In  such  cases,  it  is  possible  to  represent  some  of  the 
troublesome  structural  uncertainties  with  analytical  tricks  (Davis,  Bankes,  and  Egner,  forth¬ 
coming),  such  as  parameterizing  the  form  of  functions  and  permitting  certain  kinds  of  sto¬ 
chastic  behaviors  motivated  by,  e.g.,  historical  observation.  Even  the  directionality  of  arrows  in 
an  influence  diagram  may  be  varied  parametrically. 


Table  3.1 

Comparison  of  Courses  of  Action  Under  Uncertainty 


Course  of 
Action 

Most  Likely 
Outcome 

Best-Case 

Outcome 

Worst-Case 

Outcome 

Net  Assessment  of 
Course  of  Action3 

1 

Good 

Good 

Marginal 

2 

Good 

Very  Good 

Marginal 

3 

Very  Good 

Very  Good 

Very  Bad 

a  Depends  on  the  decisionmaker  and  the  context.  A  conservative  assessment  would 
emphasize  most-likely  and  worst-case  columns;  others  would  give  more  weight  to 
the  upside  potential.  If  the  "do-nothing"  baseline  (not  shown)  is  considered  utterly 
unacceptable  (e.g.,  by  revolutionaries),  the  upside  potential  may  be  the  key. 
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Table  3.2 

Alternative  Ways  to  Create  Simplified  Models  for  Exploratory  Analysis 


Approach 

Description 

Comments 

"Freeze"  many  variables  of  a 
more  complex  model;  conduct 
exploratory  analysis  on  a  set  of 
key  parameters3 

Find  or  create  higher-level 
parameters 

Ensure  legitimacy  of  freezing  of 
other  variables 

Feasible  only  in  some  models,  e.g., 
those  designed  as  top-down  with 
multiresolution  modeling  techniques 
(MRM).  For  others,  important 
uncertainties  are  often  too  deeply 
buried  in  databases 

Build  a  new  simple  modelb 

Approach  the  problem  top-down 
Build  in  specific  hooks  to  higher- 
resolution  models 

Parameterize  extensively 

Allow  for  structural  variations 
as  well 

Building  simple  models  is  easy;  building 
good  ones,  which  abstract  properly  and 
connect  to  more  detailed  models,  is  not 

Build  a  motivated  metamodelc 

Start  with  a  "trusted"  complex 
model 

Hypothesize  a  much  simpler  model 
that  is  still  phenomenologically 
motivated,  but  in  aggregate  terms 

Build  in  correction  factors  with 
unknown  coefficients 

The  trusted  model  may  not  include 
important  dimensions  of  uncertainty, 
such  as  adaptive  decisions.  If  trust  is 
justified,  however,  this  approach  can 
lead  to  reliable  simple-and-fast  models 
that  also  provide  good  explanations  of 
the  results 

Use  statistical  methods  to  fit  the 
postulated  model  to  the  behavior 
of  the  trusted  model 

Standard  response  surface 
methods'^ 

Fit  behavior  of  trusted  model  to 
statistical  regressions 

Problems:  trusted  model  may  not  be 
trustworthy;  also,  resulting  "simple" 
model  provides  no  meaningful 
explanation  and  may  obscure  important 
correlations  relating  to  system 
capabilities  for  systems  with  multiple 
individually  critical  components 

a  Much  work  of  this  type  has  been  done  with  JICM  (Fox,  2003). 


b  Many  examples  exist  (Davis,  Bigelow,  and  McEver,  2000;  Davis,  McEver,  and  Wilson,  2002),  although  most  were 
seen  as  one-time  actions  rather  than  methodology.  This  approach  is  also  common  in  science. 

c  This  is  a  synthesis  of  cause-effect  modeling  and  statistical  analysis  (Davis  and  Bigelow,  2003). 

d  This  approach  is  commonly  taught  in  operations  research  and  applied  statistics. 


A  complementary  approach  that  does  not  depend  on  any  single  model  is  ensemble  mod¬ 
eling  or  modeling  by  committee  (Henninger,  Pratt,  and  Roske,  2006).  The  notion  has  been 
embraced  by  the  “machine  learning”  community  (Dietterich,  2000),  which  often  uses  the 
method  to  boost  predictive  utility  of  classifiers.  The  primary  differences  among  model  esti¬ 
mates  often  reflect  differences  in  sampling  (e.g.,  bootstrapping,  bagging).  The  notion  of  ensem¬ 
bles  has  also  been  used  by  the  weather-modeling  community  (Franklin,  2005).  In  this  area,  the 
models  can  take  on  distinctly  different  forms  (e.g.,  statistical,  dynamic).  And,  finally,  the  term 
“ensemble”  is  also  used  in  studies  of  global  warming  and  long-term  futures  (Lempert,  Popper, 
Bankes,  2003). 
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Since  nomenclature  between  (and  within)  communities  can  differ,  comparisons  can  be 
somewhat  ambiguous.  That  is,  one  person’s  interpretation  of  what  can  be  parameterized  may 
intersect  with  another’s  vision  of  what  becomes  an  underlying  assumption.  Given  the  number 
of  dimensions  that  could  add  to  robustness  of  our  analyses  (e.g.,  random  seeds,  data  sources 
and  sampling  methods,  parameterization,  scenarios,  algorithmic  implementations,  underlying 
theoretical  assumptions)  coupled  with  the  imprecision  in  terms,  it  would  perhaps  be  prudent 
to  synthesize  related  information  into  a  cogent  form. 

In  ensemble  modeling,  one  starts  with  a  set  of  models,  each  of  which  has  its  own  predic¬ 
tions.  One  then  combines  results,  perhaps  with  weighted  sums  or  some  other  schema,  in  an 
attempt  to  improve  the  accuracy  of  the  overall  prediction.  The  concept  is  loosely  based  on  the 
notion  that  if  each  model  is  more  likely  to  be  right  than  wrong,  the  prediction  of  the  majority 
of  models  is  also  more  likely  to  be  right  than  wrong.  Further,  the  probability  that  the  correct 
outcome  is  supported  by  a  majority  of  the  models  should  be  a  monotonically  increasing  func¬ 
tion  of  the  number  of  the  models  in  the  ensemble,  converging  to  one  as  the  size  of  the  ensemble 
approaches  infinity.  Combining  results  of  multiple  models  is  not,  of  course,  a  substitute  for 
designing  good  models,  but  it  is  often  not  so  easy  to  know  which  of  many  competing  models  is 
better  or  worse,  in  which  case  ensemble  modeling  can  be  very  helpful.  As  a  caution,  however, 
if  the  individual  models  are  more  likely  to  be  wrong  than  random  chance,  then  combining 
results  will  worsen  prediction. 


Exploratory  Analysis  for  Portfolio  Work 

A  final  concept  to  mention  is  at  the  frontier  of  current  work:  “exploratory  analysis  of  portfo¬ 
lios.”  DoD  is  beginning  to  use  portfolio  methods  extensively  in  its  capabilities-based  planning, 
but  the  methods  and  tools  for  doing  so  are  only  emerging,12  with  current  work  being  for  the 
Office  of  the  Secretary  of  Defense  (Acquisition,  Technology,  and  Togistics).  A  recent  conclu¬ 
sion  of  that  effort  is  that  the  methods  of  exploratory  analysis  that  have  emerged  over  the  past 
decade  need  to  be  applied  now  to  portfolio  assessments  because  attempting  to  use  portfolio- 
analysis  methods  for  tradeoff  work  is  exceedingly  unreliable  without  examining  how  conclu¬ 
sions  change  as  a  function  not  only  of  scenario,  but  also  of  weighting  factors,  aggregation 
rules,  subjective  judgments  about  different  types  of  risk,  and  costs.  Because  of  the  magnitude 
of  uncertainties  and  diverse  nature  of  the  factors,  even  the  best  analysts  cannot  easily  reach 
sound  conclusions  without  more  far-reaching — and  semiautomated — exploration.  In  the  past, 
that  was  impossible,  but  it  is  now  technically  feasible.  Given  that  policymakers  are  repeatedly 
calling  for  tradeoff  analyses  within  and  across  the  new  joint  capability  areas,  the  extension  of 
exploratory  analysis  to  portfolio  work  seems  to  us  essential.  See  Table  3.3. 
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Table  3.3 

Action  Items  for  Improving  Treatment  of  Uncertainty  and  Risk 


Science  and  Technology 


Improve  or  develop  tools  to  enable  or  simplify 
exploratory  analysis 

Modify  "big"  models  (JICM,  Storm,  etc.) 

Build  new  simple  models 

Build  "motivated  metamodels" 

Improve  tools  for  analysis  of  exploratory  runs 

Improve  and  develop  methods  to  support 
integrated  families  of  models 

Develop  theory  for  cross-resolution  calibration  and 
multiresolution  modeling;  application  to  DoD's  "big-four" 
set  of  simulations 

Adapt  best  practices  from  real-options  theory 
for  defense  work  where  applicable 

Mine  recent  industry  experience  for 
insights  and  methods 

Apply  in  applicable  subdomains 

Conduct  prototype  studies  and  then 
build  appropriate  tools 

Improve  and  field  portfolio  methods  for 
capabilities  analysis 

Build  top-down  portfolio  tools  with  multiple  levels  of  zoom 

Generalize  methods  of  exploratory  analysis  for  portfolio 
work 

Within  Analytic  Studies 

Specific  studies 

Build  terms  of  reference  that  demand  analysis  of  risk 
and  uncertainty  based  on  exploration  and  evaluation  of 
candidate  flexible,  adaptive,  and  robust  strategies 

CHAPTER  FOUR 
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Understanding  the  Issues 

Although  the  conclusion  was  valid  even  in  the  allegedly  “simpler  days”  of  the  Cold  War,  the 
following  logic  seems  compelling: 

Whereas 

Today’s  special  M&S  challenges  involve  people,  decisions,  and  behaviors; 

Current  DoD  M&S  are  poor  at  representing  human  creativity,  adaptation,  and  variation; 
Other  methods,  such  as  human  gaming  and  interactive  simulation  are  better  in  this  regard; 
and 

Much  information  can  be  had  from  studying  real  wars,  operations,  tests,  and  exercises. 

M&S  for  analysis  should  be  construed  broadly  so  as  to  include  these  other  methods. 

Doing  so  will  involve  breaking  down  traditional  barriers.  What  is  needed  is  a  mix  of  ana¬ 
lytical  instruments  and  types  of  scientific  inquiry.  Such  a  mix  would  include 

1 .  diversity  of  models  (with  varied  levels  of  resolution,  perspective,  character,  and  degrees 
of  interactivity) 

2.  human  games  and  other  exercises,  structured  to  increase  rigor  and  analytical  content 

3.  laboratory  and  held  experiments 

4.  other  empirical  work,  drawing  upon  real-world  operations  planning,  training,  history 
(including  lessons-learned  studies),  and  consultation  with  experts  in  numerous  disci¬ 
plines  and  functions. 

It  is  one  thing  to  assert  that  such  a  mix  should  be  used,  but  many  questions  arise,  such  as 
whether  human  gaming  can  actually  assist  in  analysis.  Some  of  these  are  discussed  in  subse¬ 
quent  sections. 

Table  4.1  elaborates  by  contrasting  strengths  of  some  of  the  instruments.1314  At  the  top 
left,  by  simple  models  we  mean  relatively  simple  models  (which  may  also  be  simulations),  rang¬ 
ing  from  closed-form  equations  to  models  implemented  in  high-level  languages,  such  as  Micro¬ 
soft  Excel  or  Analytica.  They  are  characterized  in  Table  4.1  as  very  good  (white)  for  their  agility 
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Table  4.1 

Relative  Merits  of  Illustrative  Elements  of  a  Family  of  Tools 


Examples 


START,  CAPE,  EXHALT, 
many  unnamed 

THUNDER,  ITEM, 

JICM,  JWARS 

Enhanced  versions  of 
above 

Models  built  with 
Isaac  or  MANA 

Models  built  with 
Repast,  RAND-SEAS, 
GA-MANA 

Janus,  JCATS,  NSS 


NOTES:  The  shading  depends  on  many  implicit  assumptions.  The  key  point  is  that  instrument  classes 
have  different  virtues. 


ABMs:  agent-based  models. 

MRM:  multiresolution  modeling. 
GA-MANA:  a  model  built  with  MANA 
using  genetic  algorithms. 


Very  poor 


Very  good 


RAND  OP176-T-4.1 


and  transparency.  A  model’s  agility  can  be  measured  by  the  rime  and  effort  required  to  under¬ 
stand  the  possibility  space  that  the  model  is  intended  to  address.  Obtaining  such  a  synoptic 
view  may  require  exploratory  analysis  that  simultaneously  varies  many  input  parameters — i.e., 
something  far  more  than  usual  sensitivity  analysis.  Agility  depends  on  the  model’s  design,  the 
technology  in  which  it  is  implemented  (e.g.,  a  spreadsheet  rather  than  a  complicated  C-language 
program),  speed,  and  other  factors.  Agile  models  are  also  good  for  answering  what-if  questions 
quickly.  The  relatively  simple  models  that  we  have  in  mind  can  also  be  quite  transparent.  The 
analyst  can  fully  understand  assumptions  and  can  explain  the  analytic  story  behind  results. 
Some  of  the  very  features  that  make  simple  models  so  useful,  however,  include  a  narrowing  of 
scope  and  substantial  abstraction,  which  means  that  most  such  models  usually  say  little  about 
the  underlying  physical  or  human  phenomena,  except  perhaps  in  highly  abstract  terms,  such  as 
characterizing  an  army’s  morale  as  low  and  reducing  its  effective  strength  accordingly.15 

As  a  contrast  (bottom  right),  field  experiments  are  good  or  very  good  for  representing 
phenomenology,  including  human  issues,  but  are  poor  for  agility  and  transparency. 

To  touch  upon  the  other  instruments  in  Table  4.1,  consider  the  big  strategic  simula¬ 
tions  concerned  with  theater  and  multitheater  analysis,  primarily  at  the  theater-strategic  and 
operational  levels.  In  principle,  these  have  good  or  at  least  moderate  agility,  and  they  can  also 
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be  relatively  transparent.16  The  current  versions  of  such  models  are  typically  not  very  adap¬ 
tive,  however,  and  are  used  to  study  particular  approved  scenarios  in  depth.  If  they  were  made 
more  adaptive  (as  discussed  in  a  later  section),  given  more  multiresolution  features  as  described 
below,  and  used  for  broader  exploratory  analysis,  their  value  could  rise  a  notch  to  light  gray  or 
white. 

Simple  agent-based  models  of  the  bottom-up  variety  (e.g.,  models  based  on  MANA  or 
Pythagoras)  have  at  least  moderate  ability  (middle  gray)  to  explore  phenomenology  and  human 
action,  although  analysis  with  them  is  not  straightforward.17  In  other  respects,  agent-based 
models  (ABMs)  are  typically  quite  limited.  Much  better  agent-based  models  are  possible  and 
are  being  pursued.  These  may  include  a  mix  of  top-down  and  bottom-up  features,  more  ratio¬ 
nal  decisionmaking  based  on  balancing  considerations  through  an  objective  function,  etc. 
Detailed  models  (e.g.,  those  at  mission,  tactical,  or  engagement  levels)  are  essential  underpin¬ 
nings  even  for  higher-level  defense-planning  work  because  they  can  represent  underlying  phe¬ 
nomena  well,18  motivate  sound  aggregate  models19  (e.g.,  the  strategic  simulations),  and  some¬ 
times  be  used  to  calibrate  them.  Such  models  are  also  an  essential  link  between  the  “policy  and 
programs  world”  to  the  world  of  war  fighters  and  engineers.  Such  models,  however,  are  poorly 
suited  to  higher-level  analysis  and  decision  support. 

Human  war  gaming  is  quite  agile  in  the  sense  that  a  game  can  be  quickly  put  together  to 
deal  with  previously  unstudied  issues,  but  games  are  notoriously  deficient  in  breadth,  focusing 
on  a  single  scenario  or  a  few  vignettes.  They  are,  however,  excellent  vehicles  for  highlighting 
“real”  factors  in  the  world,  including  likely  or  possible  human  perceptions  and  behaviors.  A 
human  game,  for  example,  will  often  suggest  an  unanticipated  adversary  thought  pattern  or 
an  unanticipated  constraint  on  what  U.S.  and  allied  leaders  would  be  able  and  willing  to  sanc¬ 
tion.  Historical  work  is  not  at  all  agile,  but  it  covers  substantial  ground.  It  is  a  rich  source  of 
knowledge  about  what  “really”  happens  and  what  humans  sometimes  do  and  how  frequently 
they  make  mistakes  on  the  one  hand,  but  achieve  audacious  success  in  other  cases.20  Other 
historical  research  provides  empirical  evidence  of  aggregate-level  phenomena,  which  can  be 
powerful  in  developing  M&S.21 

Although  the  evaluations  shown  are  merely  illustrative,  depending  on  many  underlying 
assumptions,  the  general  point  is  valid:  Much  value  can  be  gained  from  drawing  on  the  mul¬ 
tiple  instruments  and  sources  of  information.  It  follows  that  analytical  organizations  should 
consciously  plan  to  have  and  exploit  the  full  range  of  instruments  and  forms  of  inquiry. 

Taking  a  Portfolio  Approach  to  Investments  in  M&S  for  Analysis.  This  reasoning  suggests 
that  in  the  years  ahead  DoD’s  analytical  organizations  should  design  their  analysis  and  supporting 
research  with  an  explicit  portfolio  approach ,  i.e.,  investing  in  a  range  of  instruments.  Figure  4.1 
characterizes  schematically  the  current  investment  portfolio  of  many  military  analytical  orga¬ 
nizations.  Our  assertion  (without  reliable  empirical  basis  and  perhaps  exaggerated  slightly22)  is 
that  the  current  de  facto  portfolios  are  extremely  distorted  in  favor  of  high-expense,  relatively 
detailed  simulations.  The  figure  also  notes  explicitly  that,  in  the  broader  view  attempted  in 
this  paper,  analysis  is  not  equivalent  to  modeling  and  simulation,  and  M&S  is  not  equivalent 
to  simulation.23  The  behavior  of  M&S  organizations,  however,  suggests  that  only  simulation 
survives  at  budget  time.24 
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Figure  4.1 

Current  Investment  Portfolios  of  M&S  Organizations 


RAND  OPT 76-4. 7 


H  Simple  models 

□  Complex  high-level 
simulations 

|]  Simple  high-res  M&S 

S  Complex  high-res 
simulations 

□  Human  war  gaming 

^  Empirical  analysis 

(history,  lessons  learned) 

|  Experiments 


That  analysis  is  not  equivalent  to  M&S,  and  certainly  not  to  simulation,  helps  to  explain 
why  there  has  sometimes  been  a  tension  between  M&S  groups  and  analysts  not  in  such  groups. 
Higher-level  analysis  depends  on  agile  work  cutting  across  boundaries  and  addressing  a  broad 
range  of  issues.  That  is  often  a  poor  match  for  data-intensive  simulation  activities,  especially 
those  that  depend  for  their  success  on  complex,  consensus-seeking,  and  lengthy  interactions 
across  multiple  organizations. 

The  good  news  is  that  the  functions  that  are  underinvested  and  underemphasized,  such  as 
agile,  broad  analysis,  are  also  less  expensive.  Achieving  balance  in  the  portfolios  would  not  be 
as  economically  disruptive  as  might  be  thought,  especially  if  some  economies  can  be  achieved 
in  big-model  work,  through  consolidation  and  more  efficient  data-handling  technology. 

Subsequent  sections  discuss  how  to  go  about  this  family-of-tools  approach,  particularly 
with  respect  to  use  of  agent-based  models,  gaming,  and  use  of  experts. 

Another  crucial  issue  is  replenishing  the  supply  of  first-rate  analysts.  Ultimately,  it  is  the 
analysts  who  determine  the  quality  of  work  done.  Even  the  best  M&S  will  produce  little  if  the 
analysts  are  inadequate.  This  subject,  which  can  be  regarded  as  one  of  human  capital,  is  beyond 
the  scope  of  the  current  study  but  badly  in  need  of  further  work  (see,  e.g.,  National  Research 
Council,  2006). 
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Agents  and  Other  Methods  for  Improving  Model  Adaptiveness 

General  Approaches 

As  discussed  earlier,  M&S  must  become  substantially  more  adaptive  if  they  are  to  be  able  to 
inform  the  choice  of  strategies  that  are  intended  to  be  flexible,  adaptive,  and  robust.  The  adap¬ 
tiveness  may  relate  to  having  (1)  submodels  that  represent  decisionmaking  by  commanders;  (2) 
submodels  that  adjust  simulated  strategy  and  tactics  depending  on  objectives,  situation,  and 
projections;  or  (3)  submodels  that  represent,  e.g.,  the  behavior  of  individuals  (perhaps  adver¬ 
sary  leaders),  groups  (for  instance,  a  particular  regional  tribe  or  the  military  establishment  of 
a  dictator  in  a  losing  war),  or  countries  (e.g.,  a  country  that  might  or  might  not  become  more 
nationalistic  and  resistant  as  the  result  of  strategic  bombing).  These  are  only  some  of  many 
examples.25 

The  methods  available  for  improving  adaptiveness  include  using  agents,  control  theory, 
game-theoretic  methods,  or  more  ordinary  model-related  operations-research  algorithms.  The 
methods  may  be  deterministic,  stochastic,  or  a  hybrid  of  the  two.  These  methods  should  be 
seen  as  “in  competition.”  Unfortunately,  the  book  has  not  yet  been  written  on  when  each  type 
of  method  is  suitable  nor  has  there  been  one  written  on  how  the  methods  can  be  related  to 
each  other. 

Agent-Based  Models 

Rooted  in  artificial  intelligence  research,  agent-based  modeling  has  been  adopted  by  the  com¬ 
plexity  sciences  because  it  shows  so  much  promise  for  dealing  with  certain  kinds  of  issues, 
whether  they  be  tactical  or  in  the  domain  of  social-cultural-military  interactions.26  Many  types 
of  agent-based  models  already  exist,  and  other  types  are  needed  to  fill  particular  gaps. 

One  distinguishing  characteristic  relates  to  whether  the  agents  are  conceived  top-down, 
as  part  of  command  and  control,  or  bottom-up,  as  part  of  atomic  entities  that  act  indepen¬ 
dently  but  generate  higher-level  emergent  behaviors  (e.g.,  riots  arising  from  the  action  of  indi¬ 
viduals  in  a  crowd,  groupings  of  tribes  into  coalitions,  or  steady-state  mass  behaviors  of  traffic 
flow).  A  second  characteristic  is  whether  the  agents  are  part  of  a  simple  or  sophisticated  larger 
simulation.  A  third  characteristic  is  whether  the  agents’  internal  structures  are  simple  (“light”) 
or  sophisticated  (“heavy”).  Traditionally,  light  structures  are  associated  with  bottom-up 
agents  populating  simple  simulations,  whereas  heavy  structures  are  associated  with  top-down 
agents  interacting  in  a  sophisticated  simulation  or  using  multiple  factors  and  correspondingly 
multifaceted  logic  to  make  decisions  or  determine  other  behaviors.  In  this  instance,  these  types 
of  agents  are  distinguished  by  the  degrees  of  competence  they  must  exhibit  when  interacting 
in  their  simulated  environments.  Further,  top-down  agents  are  also  often  used  in  “long-life” 
situations,  in  which  a  particular  agent  needs  to  behave  appropriately  and  maintain  awareness 
of  its  environment  for  a  long  period  of  time  (hours  to  days)  while  performing  many  different 
activities  during  the  span  of  its  existence.  As  the  agent-related  disciplines  mature  and  expand, 
hybrids  are  being  developed  and  all  of  these  distinctions  may  become  less  clear.27 

Many  other  characterizations  of  agents  exist.  Some  relevant  to  the  defense-planning  anal¬ 
ysis  community  relate  to  the  degree  to  which  agents  (1)  are  autonomous,  (2)  are  “social”  (inter¬ 
acting  with  each  other),  (3)  are  purposeful  rather  than  reactive,  (4)  use  decision  theoretic  logic, 
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(5)  are  adaptive  in  the  sense  of  learning,  and,  where  applicable  (as  in  agents  acting  as  decision 
models),  and  (6)  reason  in  ways  equivalent  to  humans  in  key  respects.28 

The  principal  observation  here  is  that  generalizations  are  dangerous  in  a  domain  as  rich 
and  with  as  much  potential  as  agent-based  modeling.  Further,  despite  active  and  fruitful 
research  over  the  past  decade  or  so,  the  held  is  still  young  on  the  time  scale  of  scientific  devel¬ 
opments.  Nonetheless,  many  applications  have  already  been  made  and  a  wide  variety  of  tools 
already  exist.29 

This  said,  the  question  for  this  paper  is  how  agents  can  be  used  in  analysis,  now  or  in  the 
future.  Experience  with  agents  consistently  yields  the  set  of  problems  seen  in  Table  4.2. 

As  the  communities  cross-fertilize,  some  of  these  problems  may  be  mitigated  naturally. 
For  example,  some  work30  done  in  top-down  agent  modeling  to  facilitate  explanation  could, 
theoretically,  be  applied  to  bottom-up  agents.  Cross-fertilization  itself,  however,  is  difficult 
because  of  disparities  in  terminology  and  assumptions  among  the  communities  and  because 
researchers  often  do  not  work  very  hard  to  bridge  the  gaps.  The  way  ahead  for  agent-based 
models  is  suggested  in  Table  4.3.  Here  as  elsewhere,  we  see  synthesis  efforts  as  being  especially 
suitable  for  DoD-level  funding  (e.g.,  from  the  Defense  Modeling  and  Simulation  Office — now 
called  the  Modeling  and  Simulation  Coordination  Office  [M&S  CO]). 


Using  Gaming  in  Analysis 

A  natural  question  is  whether  human  gaming  can  in  practice  be  used  to  inform  analysis.  The 
stereotype  is  that  human  games  are  idiosyncratic  to  players,  focused  on  the  playing  through  a 
single,  undocumented,  and  relatively  unstructured  scenario. 

Table  4.2 

Challenges  with  Agent-Based  Models  for  Analysis 


They  are  not  easily  generalized.  Agents  are  often  built  bottom-up  for  specific  contexts,  with  important 
J2  knowledge  deeply  buried  and  inappropriate  for  other  contexts. 

C 

^  Cause-effect  relationships  of  observed  behavior  are  difficult  to  identify  because  they  result  from  many 
<  micro-level  relationships  and  events. 

=  Similarly,  explanation  capabilities  are  often  quite  poor,  with  the  ABM  used  more  as  a  black-box  generator 
E  of  possible  behaviors  than  as  a  traditional  model. 

o 

■g  Validation  is  difficult  and  unstable  because  of  the  many  underlying  rules  and  their  interactions,  which 

no  may  change  across  problem  areas.  Also,  the  face-validity  approach  to  validation  can  fail  because  observed 

behaviors  may  not  be  intuitive  (and  may  not  even  be  valid). 
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Knowledge  acquisition  and  engineering  are  resource  intensive,  but  shortcuts  may  result  in  "brittleness." 

Few  architectures  enable  robust,  real-time  modeling  of  human  decisionmaking;  those  that  do  are  largely 
community-specific  and  require  years  of  expertise  to  master. 

Validation  becomes  more  difficult  and  more  resource  intensive  as  the  robustness  and  variability  of  the 
models  increase. 
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Table  4.3 

Action  Items  for  Analytical  Use  of  Agent-Based  Models 


Science  and  Technology 

Invest  with  portfolio  approach 

Invest  in  different  approaches  (e.g.,  top  down,  bottom  up,  hybrid), 
in  abstraction  methods  for  representing  ABM  behaviors  in  more 
analytical  models,  and  in  related  tools  (e.g.,  knowledge  acquisition 
and  explanation) 

Write  the  book 

Support  DoD-oriented  review  papers  and  books  relating  each 
method  to  various  problems  and  functions 

Validate 

Define  and  promulgate  new  concepts  of  "validation,"  such  as 
"validation  for  the  purposes  of  exploration" 

Practice 

Develop  "code"  of  best  practices  for  use  of  ABMs 

Applications 

Invest  in  diverse  types  of  applications 

Recognize  virtues  of  applications  for  discovery,  hypothesis 
generation,  rigorous  analysis,  and  the  like 

Invest  in  simulation-creation  tools 

These  are  needed  for  rapid  development  and  adaptation  of  the 
relatively  small,  specialty  models 

Enrich  classic  force-on-force  models 

Add  decision  models  to  provide  adaptivity,  probably  using  top- 
down  agents 

Enrich  treatment  of  command  and  control 
in  M&S 

Use  ABMs  to  enable  representing  core  phenomena,  such  as  shared 
awareness  and  self-synchronization 

Supplement  combat-phase  models 
with  models  of,  e.g.,  stabilization  and 
reconstruction 

Use  bottom-up  ABMs  or  possible  abstractions  for  modeling  of 
phases  in  which  emergent  behaviors  are  particularly  important 

Broaden  participation  and  sources 

Ensure  participation  of,  e.g.,  Departments  of  State  and  Commerce, 
or  of  related  experts 

Encourage  competition  and  openness 

Invest  in  both  open  and  classified  applications,  encouraging 
openness  and  peer  review 

The  Range  of  "Gaming" 

In  fact,  “gaming”  refers  to  a  broad  range  of  very  different  kinds  of  activities,  so  generalizations 
are  unwise.  In  some  cases,  gaming  is  related  to  mathematical  game  theory,  which  seeks  to  find 
optimal  strategies  for  the  protagonists.31  In  most  cases,  however,  the  gaming  refers  to  diverse 
types  of  competitive  human  play  undertaken  without  explicit  regard  for  game  theory  per  se. 

Figure  4.2  indicates  five  of  the  many  other  dimensions  along  which  types  of  gaming  can 
be  distinguished.  In  today’s  world,  most  gaming  is  actually  recreational,  much  of  it  continu¬ 
ously  online  with  very  large  numbers  of  participants.  Recreational  games  differ  in  character, 
with  some  being  “strategy”  games,  some  first-person  shooter  games,  and  still  others  that  might 
be  called  “planning  games”  (e.g.,  SimCity). 

For  the  purposes  of  this  paper,  we  are  most  interested  in  games  that  might  be  conducted 
for  the  Department  of  Defense  as  part  of  modeling,  simulation,  and  analysis  (MS&A).  Fiuman 
gaming  can  be  used  for  at  least  the  following  analytical  purposes:  (1)  discovery,  (2)  sensitiza¬ 
tion  to  issues,  (3)  concept  development,  (4)  knowledge  elicitation  (e.g.,  of  key  factors),  (5)  iden¬ 
tification  of  assumptions,  and  (6)  testing  of  hypotheses  and  strategies. 
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Figure  4.2 

Spectra  Distinguishing  Among  Characteristics  of  "Games" 
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Examples  of  such  uses  of  gaming  are  many  if  one  merely  looks.  Armies  have  long  used  war 
gaming  to  test  hypotheses  and  strategies,  and  sometimes  to  discover  new  ones.  James  Dunnigan 
and  others  drew  heavily  upon  study  of  military  history  to  create  recreational  war  games  that  were 
used  by  young  officers  well  before  today’s  era,  in  which  analogues  are  computerized  (Dunnigan, 
2003).  Even  though  systems  analysis  is  typically  associated  with  “hard  analysis,”  its  pioneers  used 
war  gaming  extensively,32  and  such  games,  although  much  less  prevalent,  remain  important  in 
RAND’s  modern-day  systems  analysis  and  policy  analysis.33  Aspects  of  the  JICM  model  (e.g., 
soft  factors  and  a  network  rather  than  a  pistons  approach)  were  motivated  by  the  strengths  of 
human  gaming  (Allen  and  Wilson,  1987).  RAND’s  “The  Day  After  .  .  .”  games  (Molander  et 
ah,  1998)  have  been  used  for  20  years  for  initial  cuts  at  serious  policy  problems,  such  as  future 
nuclear  weapons  use  and  cyberwar.  The  U.S.  Joint  Forces  Command  routinely  uses  a  model- 
game-model  approach.  And,  quite  recently,  OSD  (PA&E)  has  used  gaming  to  study  issues  in 
the  resourcing  for  and  employment  of  special  operations  forces.  In  the  academic  realm,  a  new 
discipline  called  “serious  games”  (to  distinguish  it  from  recreational  games)  has  been  emerg¬ 
ing.  Serious  games  are  defined  as  mental  contests,  played  on  a  computer  according  to  certain 
rules,  that  use  entertainment  to  further  government  or  corporate  training,  education,  health, 
public  policy,  and  strategic  communication  objectives  (National  Research  Council,  2006). 34 

These  examples  constitute  existence  proofs  of  analytical  applications,  but  much  more  can 
in  fact  be  done.  The  following  subsections  describe  three  possible  approaches. 

Using  Human  War  Gaming  Analytically 

How  to  make  human  war  gaming  more  “analytic”  deserves  an  entire  paper;  however,  one 
approach  can  be  summarized  as  follows  (see  also  Figure  4.3):35 

•  Design  the  games  as  vignettes,  with  relatively  well-described  situations. 
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•  Use  competing  teams  with  different  backgrounds  (e.g.,  from  the  United  States,  the  UK, 
Israel,  and  Poland)  to  see  diverse  tactics  and  assumptions. 

•  Encourage  teams  to  develop  explicitly  contingent  plans  (e.g.,  with  branches  and  sequels). 

•  Protect  the  teams  organizationally,  perhaps  by  embedding  them  in  independent  groups 
such  as  federally  funded  research  and  development  centers,  war  colleges,  or  U.S.  Joint 
Forces  Command. 

•  Record  planning  factors  and  reasoning  used  during  team  play,  recognizing  them  as  the 
germs  of  “models”  that  can  later  be  formalized,  whether  quantitatively  or  qualitatively. 

•  Use  red  teams,  both  to  better  appreciate  different  ways  of  assessing  the  situation  and 
defining  objectives  and  to  draw  upon  expertise  about  adversary  military  doctrine.36 

•  Follow  up  with  analysis  and  modeling. 

Analysis  should  come  both  at  the  beginning  and  at  the  end  of  the  diagram  shown  in 
Figure  4.3.  People  skilled  in  capabilities  analysis  should  design  the  war  games  and  vignettes 
to  cover  the  space  adequately  (perhaps  with  a  combination  of  experimental  design  and  M&S 
greatly  narrowing  the  number  of  cases).  Analytically  inclined  people  with  an  appreciation  of 
and  openness  toward  soft  factors  should  record  intra-game  logic  and  then  develop  summary 
qualitative  models.  Subsequently,  modelers  should  relate  the  variables  of  the  human  games  to 
variables  of  models  and  simulations. 


Figure  4.3 

A  Process  for  Using  Human  War  Gaming  Analytically 


RAND  OP1 76-4.3 
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What  is  perhaps  remarkable  here  is  that  such  an  agenda  appears  plausible.  It  does  not 
require  any  breakthroughs  in  science  or  technology.  It  merely  posits  that  good  traditional 
quantitative  analysts  bring  their  skills  to  bear  on  a  problem  that  has  often  been  studied  in  the 
past  by  historians,  political  scientists,  and  military  officers  without  much  formal  analytical 
training.37 

Using  Gaming  to  Help  Inform  Building  of  Adversary  Models 

A  different  approach  is  to  use  gaming  to  inform  and  tune  adversary  models,  starting  from  a 
theoretical  structure.  In  this  approach,  gaming  is  used  to  test  an  initial  model  and  suggest 
additional  variables.  Procedurally,  the  following  is  the  approach:38 

•  Develop  a  theory  and  structure  for  understanding  possible  high-level  adversary  decisions 
and  behaviors. 

•  Use  political-military  seminar  war  gaming  to  check  on  the  adequacy  of  the  factors  and 
structure,  and  to  test  the  theory  in  particularly  difficult  or  ambiguous  situations. 

•  Iterate  the  theory  and  use  it  to  generate  alternative  adversary  models,  each  of  which  is 
parameterized  to  reflect  inherent  uncertainties,  even  for  specified  conditions. 

•  Use  these  adversary  models  with  exploratory  analysis  to  develop  or  test  candidate  flexible, 
adaptive,  and  robust  strategies. 

Precedents  for  this  exist  at  several  levels  of  detail.  The  same  methods  can  also  be  used  to  build 
models  of  U.S.  or  third-party  decisionmaking,  or  models  for  counterterrorism  (Davis,  2006). 

Optionally  Interactive  Simulation  in  Building  War  Plans 

A  third  approach  is  optionally  interactive  simulation,  where  humans  may  be  used  to  make 
command  and  control  decisions,  such  as  shifts  in  strategy  or  commitment  of  reserves.  At 
one  time,  this  was  actually  necessary,  but  as  M&S  improved,  the  aspiration  was  to  eliminate 
human  play  so  as  to  improve  reproducibility  and  perceived  rigor.  In  some  systems,  interactiv¬ 
ity  was  explicitly  proscribed  (as  with  the  original  JWARS);  in  others,  it  was  permitted  at  some 
natural  points  (so-called  interruptible  simulations)  but  seldom  emphasized.  The  tack  suggested 
here  is  as  follows: 

•  Have  humans  sketch  out  the  “war  plans”  to  be  used  in  simulations. 

•  Represent  the  war  plans  in  the  simulation. 

•  Conduct  the  simulation  with  interruption  points  at  which  humans  review  the  situation 
and,  as  necessary,  make  adjustments  in  the  strategy  with  new  building-block  actions. 

•  Add  the  new  action  sets  or  triggering  rules  to  the  simulation. 

•  Iterate  until  automated  play  represents  well  the  strategy  and  adaptations  of  the  humans. 

•  Repeat  this  process  with  different  human  teams,  so  as  to  develop  a  set  of  alternative  strat¬ 
egies  and  building-block  actions,  for  both  red  and  blue  (to  use  the  old  bipolar  structure) 
and,  as  necessary,  for  third  parties. 
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This  mode  of  operations  has  some  associated  technical  requirements.  First,  interruptible  play 
must  be  possible.  Second,  strategies  must  be  conveniently  representable,  to  include  branches 
and  sequels  triggered  by  state-dependent  rules.  Third,  it  must  be  possible  to  construct  easily 
accessed  libraries  of  building-block  actions,  a  number  of  which  might  be  used  in  any  given 
strategy,  perhaps  conditionally.  In  past  work,  this  style  of  man-machine  operations  has  been 
found  to  be  quite  effective.39  It  appears  not  to  have  been  much  used  in  recent  years,  however.40 
That  should  change. 

Variations  and  combinations  of  this  approach  exist  at  “entity  level,”  as  in  building  adver¬ 
sary  models  using  an  interactive  approach  in  a  commercial  game  environment  (Henninger  et 
ah,  2003).  Similar  methods  could  be  used  for  defense-planning  simulations. 

Table  4.4  summarizes  way- ahead  information  for  better  use  of  gaming  to  inform  and 
supplement  M&S-based  analysis. 


Making  Better  Use  of  Experts  in  M&S 

Another  aspect  of  a  family-of-tools  approach  is  making  better  and  more  systematic  use  of 
experts.  There  is  no  dearth  of  methods  for  using  experts.  Some  of  the  many  methods  include 
Delphi  (Helmer-Hirschberg,  1967;  Linstone  and  Turoff,  2002),  the  Analytic  Hierarchy  Process 
(Saaty,  1999),  Value-Focused  Thinking  (Parnell,  2004),  Subjective  Transfer  Function  tech¬ 
niques  (Veit,  Callero,  and  Rose,  1984),  Scenario-Based  Planning  (Schwartz,  1995),  “Day  After” 
games  (Mussington,  2003),  Policy  Analysis  Market  (Looney,  Schrady,  and  R.  Brown,  2001), 
Uncertainty  Sensitive  Planning  (Davis,  2003a),  and  Assumption-Based  Planning  (Dewar, 
2003).  A  new  method  that  envisions  mixing  exploratory  analysis  with  use  of  experts  is  called 
Massive  Scenario  Generation  (Davis,  Bankes,  and  Egner,  forthcoming). 


Table  4.4 

Action  Items  for  Improving  Use  of  Gaming  in  Support  of  MS&A 


Science  and  Technology 


Write  the  book  Develop  primer  on  how  the  various  types  of  gaming  can  be  used  for  the  various  types  of 
analytic  activities,  ranging  from  discovery  to  rigorous  evaluation 

Specific  issues  Develop  and  test  methods  for  at  least  the  following:  (1)  using  human  war  gaming 

analytically,  (2)  using  seminar  gaming  to  inform  and  evaluate  adversary  models,  (3)  using 
humans  in  optionally  interruptible  simulations  to  help  develop  and  hone  adaptive  strategies 
and  related  libraries  of  building-block  actions  that  can  then  be  used  in  closed  simulation 

Technology  Evolve  technologies  in  support  of  1-3  above  (no  breakthroughs  are  necessary) 


Within  Analytic  Studies 


Specific  studies  Include  in  terms  of  references  for  major  DoD  analytic  studies  in  which  gaming  will  be  used, 
including  red  teaming  and  gaming  with  diverse  participants  (e.g.,  Israeli  or  British  officers, 
cultural  anthropologists,  and  experts  with  backgrounds  in  on-the-ground  diplomatic  and 
economic  work) 
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Nor  is  there  a  shortage  of  “platforms”  in  which  to  place  expert  knowledge  to  use  in 
models.  Examples  include  agents,  influence  nets,  rules  and  submodels  in  system  dynamics,  the 
plans  represented  in  simulations,  and  so  on. 

Typical  experience  attempting  to  elicit  expert  knowledge  has  been  less  than  satisfactory, 
however.  Some  of  the  principal  difficulties  have  been 

•  challenges  in  finding  the  experts  in  the  first  place  (if  they  even  exist  and  are  available 
easily  for  DoD  work) 

•  the  need  to  recognize  and  deal  with  the  fact  that  experts  often  have  “agendas” 

•  unfortunate  group  dynamics,  such  as  effects  of  hierarchy  and  social  context  and  the  well- 
known  “group  think”  phenomenon 

•  inappropriate  selection,  as  in  merely  inviting  the  “usual  suspects,”  or,  worse,  inviting  only 
those  expected  to  support  a  concept  being  evaluated 

•  difficult  cross-disciplinary  discussions  due  to  experts  from  diverse  disciplines  having  dif¬ 
ferent  languages,  assumptions,  and  tacit  knowledge 

•  the  insidious  tendency  for  the  organizers  of  expert  discussion,  and  perhaps  the  group  of 
experts  itself,  to  move  toward  a  best  estimate  or  consensus,  rather  than  distributions  of 
possibilities  or  exploiting  the  so-called  wisdom  of  crowds 

•  human  shortcomings  in  thinking  through  the  implications  of  complexity,  which  includes 
nonlinearity,  multiple  types  of  feedback,  concurrent  interactions  (National  Research 
Council,  1998,  p.  15) 

•  shortcomings  of  experts  in  verbalizing  their  reasoning  or  even  in  recognizing  that  a  cog¬ 
nitive  process  has  occurred  (Deutsch,  1993). 

Doing  better  is  not  necessarily  straightforward,  and  contrasting  paradigms  exist  on  how 
to  use  experts,  as  summarized  in  the  following  dichotomies: 

•  the  wisdom  of  crowds  versus  finding  the  top  expert 

•  blink  versus  think 

•  rational-analytic  versus  naturalistic 

•  multidisciplinary  versus  interdisciplinary 

•  collaboration  versus  independent  activities 

•  reasonably  like-minded  versus  creative  tension. 

The  conflicts  are  sometimes  more  apparent  than  real,  with  different  methods  being  suitable 
in  different  circumstances.  Nonetheless,  the  list  suggests  the  need  to  think  about  the  issues 
seriously. 

The  way  ahead  in  this  area  should  include  the  actions  suggested  in  Table  4.5. 
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Table  4.5 

Action  Items  for  Improving  Use  of  Experts  in  Support  of  MS&A 


Science  and  Technology 

Write  the  book 

Develop  primer  on  how  the  various  techniques  for  using  experts  apply  in  DoD  work.  Spin  off 
learning  materials  for  continuing  professional  education  for  military  officers  and  DoD  civilians 

Specific  issues 

Find  or  develop  better  techniques  for  abstracting  expert  assessments  from  groups,  without 
omitting  significant  outliers3  or  seeking  consensus,  but  while  nonetheless  eliminating 
counterproductive  "noise" 

Technology 

Review  technology  available  to  best-use  consultants  for  DoD  MS&A  purposes,  colocated  or 
distributed 

Within  Analytic  Studies 

Specific  studies 

Specify  in  terms  of  references  for  major  DoD  analytic  studies  that  experts  will  be  used  and, 
where  appropriate,  specify  the  kinds  of  expert  advice  being  sought  (e.g.,  specify  the  use  of 
independent  red  teams  and  mechanisms  to  ensure  that  their  concerns  are  both  captured  and 
assessed) 

a  Related  matters  are  discussed,  for  example,  in  the  literature  on  risk,  as  in  discussion  of  distributions  with  "thick 
tails"  (Haimes,  1998). 


CHAPTER  FIVE 

A  New  Business  Model 


Background:  A  Closed  Analytical  Culture  and  Related  M&S 

The  Analytical  Culture 

A  decade  ago,  DoD’s  analytical  culture  was  characterized  negatively  by  a  Defense  Science 
Board  study  as  in  Table  5.1.  This  description  emerged  as  a  consensus  from  a  lengthy  and  pas¬ 
sionate  panel  discussion.  Although  some  of  the  concerns  are  no  longer  as  valid  as  they  once 
were,  others  continue  to  be.  Indeed,  very  similar  concerns  were  raised  in  2006  by  participants 
in  a  workshop  (see  Appendix  B)  contributing  to  this  paper  and  featuring  people  with  recent 
industry  experience.  The  principal  point  is  that  DoD’s  analytic  culture  tends  to  be  “closed,” 
with  an  emphasis  on  “authoritative”  models  and  data  and  on  bureaucratic  consensus  building. 
Such  an  organizational  culture  tends  to  far  less  agile,  far  less  dynamic,  and  far  less  sensitive 
to  uncertainty  than  one  that  is  more  open,  competitive,  and  freewheeling.  The  recreational- 
gaming  industry  is  a  good  example  of  the  latter. 


Table  5.1 

Changes  Needed  in  DoD  Analysis  Culture,  as  Seen  in  1996 


What  Is 

What  Should  Be 

Closed 

Open 

Bureaucratic  review 

Peer  review 

"Accredited"  analysis 

Competitive  analysis 

Model  orientation 

Subject  orientation 

Mechanical 

Meaningful 

Data  poor 

Data  rich 

Rigid  approvals 

Learning  and  adaptation 

Stable  algorithms 

Unstable  phenomena 

Suppression  of  uncertainty 

Illumination  of  uncertainty 

Suppression  of  risk 

Illumination  of  risk 

Narrow  Cold  War  style 

Oriented  to  now  and  future 

Few  accredited  scenarios 

Many  political-military  scenarios 

Point  assumptions 

Exploratory  analysis 

SOURCE:  Adapted  slightly  from  Defense  Science  Board  (1996). 
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There  are  advantages  and  disadvantages  to  each  of  the  styles,  but  the  nature  of  military 
developments  and  the  explicit  demands  of  policymakers  on  the  analytic  community  strongly 
suggest  the  need  for  a  strong  dose  of  the  features  in  the  right  column  of  Table  5.1. 

M&S  Technology  and  Relationship  to  Culture 

Contributing  to  the  analytical  culture  is  the  nature  of  DoD’s  very  approach  to  M&S,  i.e.,  its 
implicit  business  model.  The  history  of  DoD  M&S  in  support  of  higher-level  analysis,  such  as 
force  planning,  has  revolved  around  monolithic  models.  Such  models  were  designed  to  be  used 
as  a  whole.  To  be  sure,  even  early  models  were  designed  with  a  clear  concept  of  parts,  such  as 
air-to-air,  ground,  and  naval  combat  (Hughes,  1989),  and  programmers  intimately  familiar 
with  the  models  can  turn  portions  on  or  off,  or  make  changes  to  parts  without  too  much  effort. 
However,  “mere  mortals”  often  cannot.  Furthermore,  the  big  models  have  been  developed  with 
relatively  low-level  computer  languages  making  it  impossible  for  typical  analysts  to  make  quick 
changes  themselves.  The  results  are  big  models  and  analytical  processes  that  have  been  slow  to 
change. 


Improving  Agility  (and  Quality)  with  Modern  Technology 

Before  proceeding,  we  note  that  there  are  advantages  to  monolithic  designs  and  low-level  pro¬ 
gramming  languages.  They  can  be  efficient,  and  the  pieces  have  been  designed  and  mutually 
tuned  to  work  well  together,  at  least  for  the  domain  of  cases  anticipated  by  their  designers. 
They  are  “integrated”  in  the  old  sense  in  which  that  term  was  once  used,  and  much  good  work 
can  be  done  with  them.  Nonetheless,  modern  theory  and  practice  for  design  and  develop¬ 
ment  of  both  software  and  models  emphasize  modularity,41  by  which  we  mean  strict  technical 
modularity.  Individual  modules  are  developed  and  tested  separately.  Various  combinations  are 
then  assembled  as  needed,  interacting  only  through  well-defined  interfaces.  The  ability  to  do 
this  is  often  called  composability  or  component-based  design.42  Object-oriented  modeling  is 
one  example  with  a  substantial  record  of  success  over  the  past  decade  or  so.43 

The  advantages  to  such  modularity  are  immense  and  affect  quality,  testing,  comprehen¬ 
sibility,  and  maintainability.  From  the  viewpoint  of  this  analysis-centric  paper,  some  of  the 
principal  advantages  are  these: 

•  Flexibility  and  adaptiveness.  A  new  module  can  be  specialized  for  a  particular  problem  and 
swapped  in.  Modules  of  different  resolution  can  be  prepared  (and  even  included  in  the 
program)  so  that,  at  run  time,  analysts  can  choose. 

•  Continuous  improvement.  Modular  approaches  enable  continuous  improvement  as  more 
knowledge  becomes  available.  Making  changes  does  not  require  exquisite  knowledge  of 
the  overall  M&S  in  which  the  module  appears. 

•  Cooperation.  Cooperation  can  be  achieved  across  organizations,  with  different  groups 
addressing  different  modules  or  different  approaches  to  the  same  module.  Some  modules 
may  be  reused. 
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•  Competition.  For  similar  reasons,  competitions  can  be  held  comparing  alternative  designs, 
with  either  the  “best  of  breed”  emerging  or  with  alternative  versions  coexisting  for  spe¬ 
cialty  purposes. 

The  above  items  relate  to  modularizing  a  “big”  model.  However,  true  modules  are  also 
models  in  themselves;  they  can  be  used  separately,  without  the  complications  and  overhead  of 
the  entire  system.  Furthermore,  the  modules  can  be  reprogrammed  in  higher-level  languages 
(e.g.,  Visual  Basic,  Excel,  or  Analytica),  creating  stand-alone  specialty  models  that  are  very 
accessible  to  analysts  and  also  changeable  by  those  same  analysts. 

Since  so  many  current  problems  of  defense  planning  relate  less  to  sizing  total  joint  forces 
for  simultaneous  big  wars  than  to  making  good  choices  about  options  within  various  joint 
capability  areas  or  investment  choices  across  capability  areas,  analysis  may  often  be  best 
accomplished  with  specialty  models  and  combinations  thereof,  rather  than  by  using  the  big 
simulations. 

Opinions  and  tastes  differ  on  this  matter,  but  increasing  the  relative  emphasis  on  such 
specialty  models  would  likely  clarify  and  sharpen  the  key  analytical  issues.  Ideally,  these  should 
have  very  clear  relationships  to  modules  of  the  “big”  models  so  that  comparisons  of  competing 
analyses  would  remain  straightforward.44 

What  does  a  “clear  relationship”  mean?  It  might  be  convenient  to  use  modules  of  a  big 
simulation  as  stand-alone  models  (e.g.,  of  strategic  mobility  or  close-range  ground  combat). 
Doing  so  requires  a  separate  input-output  interface,  but  the  algorithms  and  data  structures  are 
identical  to  those  of  the  relevant  module  in  the  larger  simulation.  Had  the  big  simulation  been 
built  with  this  in  mind,  the  user  could  instead  use  just  those  parts  of  the  big  simulation  as  are 
needed  and  work  from  a  simplified  interface  consistent  with  the  more  comprehensive  one. 

Suppose,  however,  that  the  analyst  does  not  want  to  be  bothered  with  the  complexity 
of  understanding  the  big  model  or  how  to  use  it  like  a  maestro.  Suppose  that  he  wants  to  do 
some  of  the  model  runs  himself  and  have  complete  control  and  understanding,  but  of  a  rela¬ 
tively  narrow  problem,  such  as  how  best  to  achieve  a  particular  capability,  for  example,  air- 
to-ground  interdiction  or  persistent  surveillance  over  ground  forces.  Such  an  analyst  might 
prefer  a  specialty  model,  perhaps  built  with  Microsoft  Excel  or  Analytica  (used  extensively  by 
both  RAND  and  MITRE  in  higher-level  studies).  The  relevant  algorithms  and  data  structures 
might  be  entirely  consistent  with  those  of  the  big  model;  might  be  somewhat  different,  but  in 
ways  not  fully  understood;  or  might  be  optionally  different  in  known  and  precisely  described 
ways.  Ideally,  the  latter  circumstance  would  apply,  but  that  does  not  come  about  naturally, 
because  analysts  are  typically  more  concerned  about  their  studies  than  keeping  track  of  pre¬ 
cisely  how  their  tools  relate  to  other  tools.  To  reinforce  the  point,  it  is  often  the  case  that  the 
analysts  using  specialty  models  are  able  to  do  very  insightful  and  compelling  analysis  quickly, 
without  the  overhead  of  a  big  model  that  does  not  even  include  some  of  the  considerations 
the  analyst  feels  are  important  (they  could  be  added,  but  only  by  negotiating  with  the  model’s 
developers,  keepers,  and  perhaps  some  configuration-control  board). 

We  have,  then,  a  classic  conflict.  On  one  side  are  those  who  emphasize  standardization 
using  authoritative  models  and  databases  and  configuration  control.  They  know  that  such 
things  are  very  important  in  joint  work,  so  that  all  participants  in  the  analytic  wars  are  on  the 
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same  playing  field,  and  in  which  it  should  be  difficult  for  losers  to  obfuscate  results  by  attack¬ 
ing  the  model  or  data.  On  the  other  side,  we  have  those  who  wish  to  be  agile,  unencumbered 
by  big-model  overhead,  problem  focused,  and  unburdened  by  what  they  see  as  excessive  con¬ 
cern  for  consensus  and  excessive  devotion  to  standard  but  unreliable  data  and  assumptions, 
many  of  which  obscure  important  uncertainties.  Some  individuals  work  both  sides  of  the 
divide,  from  time  to  time,  but  most  do  not. 

Our  recommendation  is  to  improve  the  openness,  dynamism,  quality,  and  agility  of 
DoD’s  analysis  by  applying  related  practices  to  M&S: 

•  Improve  the  modularity  and  “composability”  of  the  department’s  key  big  models,  so 
that  they  can  be  improved  and  adapt  more  rapidly  and  benefit  from  the  marketplace  of 
ideas.45 

•  Explicitly  increase  investment  in  smaller,  specialized  models  supporting  agile  analysis, 
and  in  the  tools  and  techniques  for  developing  them  rapidly.46 

Interestingly,  both  of  these  could  be  accomplished  simultaneously,  to  the  extent  that  the  big- 
model  modules  are  viewed  in  parallel  with  the  separate  models.  That  is,  someone  with  a  good 
specialized  model  should  be  able  to  reprogram  it  so  that  it  can  be  a  replacement  module  of  the 
large  model.  And,  conversely,  a  good  large-model  module  can  spin  off  a  self-contained  spe¬ 
cialized  model  (or,  perhaps,  be  used  within  the  big-model  structure  by  turning  off  and  hiding 
unused  modules).  It  should  certainly  be  possible  to  develop  routine  procedures  for  comparing 
assumptions  and  results  of  alternative  models.  Thus,  our  third  admonition  is 

•  As  a  somewhat  lesser  priority,  invest  in  developing  easy  and  well-oiled  procedures  for 
comparing  alternative  models  and  their  results — not  so  as  to  achieve  equivalence,  but  so 
as  to  understand  differences  (and  be  able  to  achieve  equivalence  in  well-defined  limits). 

Given  the  concerns  of  policymakers  about  the  inadequacy  of  current  M&S  for  the  new  chal¬ 
lenges,  the  highest  priority  should  be  on  the  first  two  items  above. 
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The  Evolution  of  Analysis  M&S 

Policymakers  have  identified  new  challenges  for  analysis  in  support  of  defense  planning,  which 
in  turn  has  generated  new  requirements  for  M&S.  As  the  way  ahead  is  pondered,  it  is  useful 
as  well  to  note  where  we  have  been,  where  we  are  now,  and  where  we  want  to  be.  Figure  6.1 
provides  such  a  picture.  Although  only  notional  and  schematic,  with  no  firm  basis  in  empirical 
work,  the  picture  conveyed  is  thought  to  be  accurate. 
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The  first  point  is  that  M&S  support  for  defense  analysis  has  improved  substantially  over 
the  past  15  years.  Comparing  the  situation  today  with  that  in  1990,  we  see  far  more  jointness, 
carefully  integrated  studies  with  substantial  richness,  and  a  common  baseline  of  standard  sce¬ 
narios  and  assumptions  on  the  basis  of  which  to  do  analysis  of  alternatives.47  Today’s  analysis  is 
quite  good  at  representing  higher-level  maneuver  of  large  forces.  We  also  see  improvements  in 
treating  uncertainty,  “soft  factors,”  the  concept  of  “effects,”  and  command  and  control.  As  the 
figure  suggests,  however,  much  more  should  be  demanded  over  the  years  ahead,  much  of  it  in 
the  near  to  mid  term,  since  no  technological  breakthroughs  are  necessary. 

Jointness  and  common  baselines  should  improve  further  and,  importantly,  be  achieved 
with  far  greater  efficiency.  Representation  of  maneuver  will  need  to  be  better  for  the  increas¬ 
ingly  nonlinear  battlefield  and  for  the  world  in  which  important  maneuvers  occur  with  smaller, 
more  agile,  highly  coordinated  forces,  sometimes  deep  into  enemy  territory,  and  often  in  the 
context  of  highly  irregular  combat. 

Particularly  relevant  to  the  new  challenges,  however,  tomorrow’s  M&S  must  include 
highly  adaptive  strategies  because  they  will  be  used  to  evaluate  alternative  flexible,  adaptive, 
and  robust  (FAR)  strategies.  Such  evaluation  will  require  testing  across  a  broad  possibility 
space  and  M&S  that  represent  the  planned  and  unplanned  adaptations  possible  under  the 
different  strategies.  This  implies  a  far  more  extensive  approach  to  uncertainty  analysis,  one 
characterized  by  true  exploratory  analysis  not  only  across  diverse  name-level  scenarios  (e.g., 
crisis  with  Iran),  but  within  each  such  scenario.  Treatment  of  soft  factors  should  be  routine 
and  related  sensibly  to  underlying  factors.  Treatment  of  effects  should  not  only  be  routine  and 
extensive,  but  should  also  provide  sound  information  on  the  range  and  rough  distribution  of 
possible  effects — including  those  that  are  counterproductive  or  better  than  anticipated.  Future 
M&S  should,  where  needed,  be  exceptionally  good  at  representing  network- centric  planning 
and  operations. 


Summary  Recommendations  on  Investment 

Many  of  the  recommendations  and  suggestions  above  relate  more  to  emphasis  and  manage¬ 
ment  than  to  DoD-level  investment.  Table  6.1  instead  focuses  exclusively  on  DoD-level  invest¬ 
ments.  These  are  seen  as  common  goods  particularly  worthy  of  DoD-level  attention,  whereas 
the  vast  bulk  of  M&S  work — even  that  relevant  to  DoD  analysis — is  and  should  be  conducted 
by  the  various  services  and  defense  agencies.  Table  6. l’s  recommendations  fall  into  three  cat¬ 
egories,  consistent  with  the  DoD  priorities  discussed  earlier.  Within  each  category,  the  table 
indicates  desirable  policy  actions — essentially  the  “spin”  to  be  applied  in  expressing  require¬ 
ments  and  writing  terms  of  reference;  investments  and  their  priority  (1,  2,  or  3)  within  the  cat¬ 
egory;  and  suggestions  as  to  whom  the  sponsor  might  be  and  the  amount  of  investment  that 
might  be  needed. 

Note  that  all  categories  have  the  same  overall  priority.  Categories  2  and  3  relate  spe¬ 
cifically  to  many  of  the  “new”  DoD  leadership  demands  and  priorities,  but  category  1  relates 
broadly  to  the  core,  continuing  DoD  work  in  planning  its  programs  and  balancing  its  associ¬ 
ated  portfolios  of  investments. 
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Table  6.1 

Summary  Recommendations  for  DoD-Level  Investment 
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Some  words  are  appropriate  regarding  the  estimates  of  cost.  It  would  be  very  easy  to 
expand  ambitions  and  stimulate  large,  expensive  software-development  activities — not  on  the 
scale  of  JSIMS  and  JWARS,  but  on  the  scale  of  tens  of  millions  of  dollars.  That  is  not  what 
we  suggest.  Instead,  the  recommendations  focus  on  a  combination  of  (1)  consolidation  activi¬ 
ties,  (2)  work  on  critical  enablers,  (3)  “sense-making”  research  to  make  sense  out  of  what  often 
seems  chaotic  and  unplanned,  (4)  common-good  systematic  research,  and  (4)  fresh  design 
work. 

To  put  it  differently,  Table  6.1  omits  many  high-priority  investment  items,  e.g.,  advanced 
agent  methods  to  assist  in  modeling  and  analysis  of  culture-sensitive  issues,  only  because  they 
are  apparently  being  pursued  by  the  services,  commands,  and  defense  agencies.  OSD  should 
ensure  that  this  is  true  and,  if  it  is  not,  arrange  for  changed  priorities.  We  mention  this  because, 
although  a  good  deal  of  related  research  has  been  funded  in  recent  years,  quality  and  results 
have  been  predictably  varied.  A  worry  is  that  funding  organizations  would  back  away  from  the 
new  and  difficult  subjects  requiring  fundamental  rethinking  and  experimentation.  Instead, 
continued  investment  is  badly  needed,  although  more  coherence  and  emphasis  of  synthesis 
should  also  be  sought.  Much  of  the  work  needed  is  in  the  province  of  science  and  technology 
and,  thus,  the  province  of  the  Director  for  Defense  Research  and  Engineering. 

Within  the  first  action  class  of  Table  6.1  (routine  treatment  of  uncertainty),  the  first 
recommendation  is  complex,  calling  for  improvement  of  big  models  and  support  of  simpler 
models  (often  specialty  models)  and  assuring  the  ability  to  compare  the  models.  This  recom¬ 
mendation  requires  special  discussion  because,  taken  as  a  whole,  it  corresponds  to  a  distinctly 
new  business  model  for  DoD-level  investment.  As  a  whole,  the  need  exists  for  greater  open¬ 
ness,  competition,  agility,  and  capabilities.  Past  investments  have  tended  to  emphasize  stan¬ 
dardization,  and  the  predictable  result  has  been  to  generate  models  and  modeling  activities 
that  are  good  at  what  they  do,  with  consistent  data,  but  that  lack  the  vigor,  creativity,  and 
cross-cutting  flexibility  that  are  needed.  DoD  should,  on  the  one  hand,  integrate  and  improve 
its  current  strategic  models,  make  the  result  as  modular  as  possible,  and  actively  plan  con¬ 
tinuing  competition  and  evolution  at  the  module  level.  Different  module  versions  will  likely 
be  superior  in  different  analyses,  whether  because  of  resolution  or  which  phenomena  need  to 
be  represented  well.  Some  module  versions  might  allow  for  optional  human  play  or  might  be 
tuned  to  recent  empirical  information.  DoD  should,  on  the  other  hand,  also  encourage  and 
use  more  relatively  small  specialty  models,  whether  for  agile  aggregate-level  work  or  for  very 
narrow  analysis  of  mission  capabilities — analysis  unencumbered  by  the  weight  of  full  cam¬ 
paign  models.  DoD  should  also  invest  modestly  to  ensure  that  researchers  are  able  readily  to 
compare  assumptions  and  results  of,  e.g.,  the  specialty  models  and  relevant  big-model  mod¬ 
ules.  In  some  cases,  a  specialty  model  may  be  identical,  except  for  programming  language, 
to  a  big-model  module;  in  other  cases,  there  may  be  important  and  desirable,  but  optional, 
differences.  Similarly,  cross-comparison  mechanisms  should  exist  to  ensure  that  DoD’s  inte¬ 
grated  campaign  model  has  understood  relationships  with  relevant  service-level  models. 

This  recommendation  is  complex,  but  we  urge  honoring  its  complexity  rather  than  invest¬ 
ing  only,  for  example,  in  the  consolidation  and  integration  of  current  strategic  models  for  the 
sake  of  greater  efficiency  and  standardization. 


Conclusions  and  Investment  Recommendations  37 


The  second  recommendation  in  this  action  class  is  to  generalize  the  concepts  of  “explor¬ 
atory  analysis”  for  effective  application  in  portfolio  analysis,  e.g.,  analysis  to  inform  resource 
allocation  within  a  capability  area  or  across  multiple  capability  areas.  The  allure  of  portfolio 
methods  is  considerable,  but  the  results  are  often  bogus  because  the  results  depend  too  sensi¬ 
tively  on  a  myriad  of  low-level  assumptions.  As  with  capabilities  analysis  generally,  the  solution 
is  to  seek  flexible,  adaptive,  and  robust  strategies  of  investment,  which  requires  exploratory 
analysis  in  which  the  assumptions  affecting  the  resource-allocation  decisions  are  systematically 
varied. 

Within  the  second  action  class  (routine  treatment  of  complex  human  and  social  issues), 
our  primary  recommendation  is  for  DoD  to  develop  an  integrative  primer  on  the  use  of  meth¬ 
ods,  such  as  red  teaming,  war  gaming,  and  use  of  experts.  This  is  a  class  of  issues  that  is  ripe 
for  coherent  and  meaningful  synthesis,  which  is  not  an  effort  for  a  committee.  We  also  men¬ 
tion  the  need  to  revise  policies  on  verification,  validation,  and  accreditation  (VV&A)  so  that 
they  acknowledge  and  deal  sensibly  with  the  vast  array  of  models  (and  associated  data)  that  is 
simply  not  “predictive”  in  the  classic  sense,  but  rather  are  intended  for  more  exploratory  work 
amid  uncertainty. 

Within  the  third  action  class  (adaptiveness  and  networking),  we  again  recommend  a  syn¬ 
thetic  activity,  starting  with  one  that  pulls  together  methods  such  as  agent-based  modeling, 
control  theory,  and  game  theory  so  as  to  clarify  which  methods  for  providing  adaptiveness  are 
most  suitable.  We  also  recommend  recognizing  that  a  new  generation  of  strategic/operational- 
level  model  is  needed  for  the  network-centric  era.  Even  JWARS  (now  JAS)  has  only  some  of 
the  features  needed,  and  we  do  not  believe  that  any  consensus  yet  exists  about  what  a  new- 
generation  model  might  look  like.  This  suggests  the  need  for  preliminary  research  and  a  com¬ 
petition  of  ideas.  This  is  distinct  from  and  in  addition  to  network-centric  models  necessary  for 
design  and  evaluation  of  particular  systems  of  systems,  which  are  of  great  importance  within 
the  acquisition  community. 
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"Analysis  Practices  Working  Group,"  March  14-16,  2006 


A  workshop  was  held  at  the  Institute  for  Defense  Analyses,  March  14-16,  2006,  organized  by 
the  Institute  for  Defense  Analyses  (IDA)  and  representatives  from  RAND  and  MITRE.  One 
of  the  working  groups  (Working  Group  3),  which  we  chaired,  focused  on  analysis  practices.48 
Attendance  was  by  invitation  only,  with  group  members  nominated  by  their  respective  orga¬ 
nizations.  Nineteen  people  (Table  A.l)  participated  throughout;  some  additional  participants 
were  able  to  sit  in  during  portions  of  the  proceeding. 

An  important  aspect  of  the  working  group’s  effort  was  to  take  an  unrelenting  analysis¬ 
centric  view,  rather  than  being  concerned  too  early  and  exclusively  with  particular  modeling 
and  simulation  issues.  Two  expert  panels  provided  brief,  but  very  candid  and  useful,  sugges¬ 
tions  from  the  perspective  of  analysis  organizations  to  which  modeling  and  simulation  are 
merely  tools  along  the  way.  These  suggestions,  then,  presented  a  view  of  “demand,”  above  and 
beyond  the  useful  material  in  the  2006  Quadrennial  Defense  Review  and  other  formal  docu¬ 
ments.  In  addition,  a  few  individuals  were  asked  to  give  short  presentations  on  specific  topics 
of  interest:  U.S.  JFCOM’s  recent  experience  with  a  combination  of  modeling  and  war  gaming, 


Table  A.l 

Members  of  Analysis  Practices  Working  Group  Panel 


Paul  K.  Davis,  Chair  (RAND) 

Amy  Henninger,  Cochair  (IDA) 
Sheila  Bankes  (Calculated  Insight) 
Chris  Chartier  (Nil) 

Ed  Crowder  (A-9) 

Augie  Fucci  (DAMO  Cl) 

Jeff  Hamman  (JHU  APL) 

Eric  Johnson  (JDS) 

Rick  Kass  (JFCOM  J-9) 

Bill  Key  (TRANSCOM  J-5) 


Andy  Kourkoutis  (J-8) 

Mary  McDonald  (N-81) 
Jimmie  McEver  (EBR) 

John  Robertson  (CAA) 

Vince  Roske  (IDA) 

Scott  Schutzmeister  (AMSO) 
Stuart  Starr  (Consultant) 

Bill  Woodson  (U.S.  Marines) 
Derk  Wybenga  (J-4) 


NOTE:  See  the  Abbreviations  list  for  the  affiliations. 
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lessons  learned  from  NATO’s  “Code  of  Best  Practices”  for  command  and  control  analysis, 
OSD’s  use  of  human  gaming  to  represent  the  challenges  of  special  operations  forces,  and  new 
tools  for  social-cultural  modeling,  simulation,  and  database  development.  Table  A. 2  lists  the 
presenters. 


Table  A. 2 

Presenters  at  Working  Group  3 


Presenter 

Affiliation 

DoD  Panel 

Eric  Coulter 

OSD  (PA&E) 

David  Alberts 

OSD  (Nil) 

Andrew  Hoehn 

RAND  Corporation  (previously  OUSDP) 

Brigadier  General  (S)  Woodward 

J-8 

Services  Panel 

Jackie  Henningsen 

Air  Force  A-9 

Robbin  Beall 

Navy  N-81 

Mike  Bailey 

U.S.  Marine  Corps  (MCCDC) 

Mike  Moore 

Short  Briefings  and  Subjects 

Stuart  Starr  (NATO  Code  of  Best  Practices) 

Consultant 

Rick  Kass  (JFCOM  Wargames  and  M&S) 

U.S.  JFCOM 

Mike  Ottenberg  (Gaming  of  SOF  [Special 
Operations  Forces]  Issues) 

OSD  (PA&E) 

Sue  Numrich  (Modern  Tools  for  Social-Cultural 
Analysis) 

Consultant 

NOTES:  PA&E:  Program  Analysis  and  Evaluation;  Nil:  Networks  and  Information  Integration; 
OUSDP:  Office  of  the  Under  Secretary  of  Defense  for  Policy  [could  be  abbreviated  OSD(P)]; 
MCCDC:  Marine  Corps  Concept  Development  Center. 
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"Methods  Working  Group,"  June  26-27,  2006 


A  workshop  was  held  at  the  Institute  for  Defense  Analyses,  June  26-27,  2006,  the  purpose  of 
which  was  to  obtain  industry  responses  to  the  challenges  identified  in  earlier  work  under  the 
project.  There  was  special  interest  in  augmenting  the  challenges  as  necessary  and  hearing  about 
potential  “solutions”  in  meeting  those  challenges.  The  methods  working  group  was  by  invita¬ 
tion  only.  Fourteen  people  participated  throughout  (Table  B.l),  of  which  eight  made  presenta¬ 
tions.  Discussion  was  extensive  and,  as  in  the  earlier  workshop,  took  an  unrelenting  analysis¬ 
centric  view,  rather  than  focusing  on  particular  M&S  per  se. 

Table  B.l 

Participants  in  the  "Methods  Working  Group" 

Panel  Members  Presentations 

Paul  K.  Davis,  Chair  (RAND) 

Amy  E.  Henninger,  Cochair  (IDA) 

Jason  Dechant,  Recorder  (IDA) 

Gerald  Bracken  (IDA) 

Mark  Altaweel  (Argonne  National  Labs) 

Michael  Anderson  (Boeing) 

Steve  "Flash"  Gordon  (GTRI) 

C.R.  Krieger  (DRC) 

Dell  Lunceford  (Total  Immersion) 

Scott  Matthews  (Boeing) 

Jonathan  Prescott  (Metron) 

James  Saultz  (Lockheed  Martin) 

Rich  Wicker  (SPA) 

Leah  Zimmerman  (Green  Manor  Group,  LLC)  Cultural  Demographics  for  Better  Decisions 

NOTES:  Company  abbreviations:  GTRI:  Georgia  Tech  Research  Institute;  DRC:  Dynamics  Research  Corporation; 

SPA:  Systems  Planning  and  Analysis. 


A  Case  for  Using  Ensembles  in  Analyses  for  Military 
Decisionmaking 


Development  of  Modeling  and  Simulation  Applications  in 
Support  of  National  and  Homeland  Security 

Boeing  Approaches:  Tools  and  Data  Focus  Challenges 

Missions  and  Means  Framework  Integration  and  Transition 

Oh  My  God,  Not  Another  DoD  "M&S-What  Should  We  Do" 
Workshop 

Characterizing  Uncertainty  and  Risk  (tool) 

Proprietary  presentation 
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APPENDIX  C 


Illustrative  Contrast  of  Questions  for  Analysis  from  the  Previous 
Era  and  Today 


Table  C.1 

A  Contrast  of  New  and  Old  Questions  for  Analysts 


Old  Questions 


New  Questions 


Force  Structure 


How  much  force  structure  do  we  need  for 
defense  of  Europe,  Korea,  and  Southwest  Asia? 

Is  defense  feasible  in  the  Persian  Gulf,  against 
regional  powers  or  against  the  Soviet  Union? 
What  kinds  of  forces  and  strategy  would  be 
needed? 

How  should  we  structure  the  "total  force" 
with  a  combination  of  active,  reserve- 
component,  and  national-guard  forces? 

What  mix  of  capabilities  do  we  need  to  cost- 
effectively  achieve  the  "best-estimated"  needs 
for  stopping  a  large  invasion  (through  attrition 
of  the  invading  army)? 


How  much  force  structure  do  we  need,  given  worldwide 
interests  and  commitments,  the  global  war  on  terrorism,  and 
diverse  missions  and  complex  contexts? 

How  do  we  balance  investments  across  missions  involving 
rogues,  failed  or  failing  states,  regional  peers,  and  the  terrorist 
threat?  And  across  activities  such  as  combat  and  stabilization? 

What  "total  force"  should  we  have  for  the  future,  given  the 
range  of  missions  and  scenarios,  and  the  now-demonstrated 
need  for  and  necessity  of  using  national-guard  and  reserve 
forces  in  combat? 

What  building-block  capabilities  would  cost-effectively 
maximize  a  future  joint  commander's  ability  to  achieve  desired 
effects  involving  enemy  top  leaders,  leadership  levels,  rank- 
and-file  military  personnel,  the  population  as  a  whole,  and 
specific  populations  in  particular? 


Approach  to  Military  Operations 


How  can  we  assess  the  concept  known  as  Air- 
Land  Battle? 

How  do  we  fold  in  NATO-alliance  issues  as  we 
conduct  defense  planning  relevant  to  Europe 
and  other  regions? 


How  do  we  assess  effects-based  operations,  rapid  decisive 
operations,  and  information  operations?  How  do  we 
characterize  and  minimize  risk? 

How  do  we  orchestrate  all  relevant  military  instruments,  and 
indeed  all  instruments  of  national  and  allied  powers,  to  achieve 
desired  effects  at  the  level  of  individuals,  small  groups,  and 
national  populations? 


Modernization 


What  could  be  accomplished  with 
"stealthy"  aircraft  that  would  be  essentially 
undetectable  by  traditional  radars? 

Can  we  count  on  high-tech  precision  weapons 
and  how  many  of  them  should  we  buy? 

What  mix  of  deep-attack  precision  weapons 
would  be  optimal  (for  specific  war  scenarios)? 


Can  we  build  adequately  survivable  and  otherwise  capable 
aircraft  for  countering  maneuver  of  relatively  large  ground- 
force  units  and  mobile  missiles  far  from  the  shore  and  far  from 
secure  airfields? 

How  do  we  assess  networked  capabilities?  Can  we  count  on 
their  effectiveness?  How  much  redundancy  is  needed? 

How  do  we  make  tradeoffs  involving  C4ISR,  involving 
networking,  systems  of  systems,  information  operations,  and 
traditional  force  capabilities? 
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Endnotes 


This  paper  is  not  intended  as  a  literature  review,  but  the  following  endnotes  provide  numerous 
pointers  to  the  relevant  literature.  We  have  emphasized  citations  that  are  readily  available  as 
full  papers  or  books.  Many  of  the  papers  are  available  online. 


Summary  Endnotes 

1  MATLAB®  is  a  registered  trademark  of  The  MathWorks  Inc. 

2  JWARS:  Joint  Warfare  System;  JAS:  Joint  Analysis  System. 


Endnotes 

1  Separate  white  papers  have  been  prepared  on  modeling  of  traditional  warfare  (Bracken  et  ah,  2006),  non- 
traditional  warfare  (Dechant  et  al.,  2006),  technology  for  M&S  (Banks  et  ah,  2006),  and  data  issues  in  M&S 
(Data  Working  Group,  2006). 

2  See  also  a  recent  National  Academy  report  (National  Research  Council,  2006).  Some  of  the  material  in  this 
paper  overlaps  with  the  senior  author’s  contributions  to  that  report.  The  NRC  study  also  includes  a  chapter  on 
Education,  Training,  and  Professional  Practice,  as  well  as,  e.g.,  more  discussion  of  social  modeling,  composabil- 
ity,  and  “serious  games.” 

3  Some  effects-based  operations  (EBO)  issues  have  been  discussed  in  the  public  domain  (Deptula,  2001;  Davis, 
2001;  Smith,  2003;  Smith,  2006).  A  web  link  to  many  sources  is  http://www.saunalahti.fi/fta/EBO.htm  (as  of 
December  18,  2006).  Effects-based  operations  are  controversial  in  some  respects  (Davis  and  Kahan,  forthcom¬ 
ing;  Jobbagy,  2006),  particularly  with  regard  to  what  is  and  is  not  new,  terminology,  whether  “predictiveness”  is 
an  appropriate  goal  for  effects-based  operations  analysis,  and  the  processes  appropriate  for  implementing  them. 
The  core  ideas,  however,  are  sound,  and  involve  worrying  about  finding  actions  that  will  improve  the  odds  of 
achieving  desired  PMESII  effects,  whether  direct  or  indirect,  and  of  avoiding  undesirable  effects.  Doing  so  often 
requires  a  complex  “system  view.” 

4  Complex  adaptive  systems  research  has  burgeoned  over  the  past  15  years,  and  many  good  sources  exist — 
some  popular  (Waldrop,  1992),  some  relatively  nontechnical  but  solidly  based  in  science  (Holland  and  Mim- 
naugh,  1996),  some  more  technical  (Bar-Yam,  2003),  and  some  applied  to  military  problems  (Ilachinski,  2004; 
Grisogono,  2006b). 

5  A  number  of  good  references  exist  on  network-centric  operations  (Cebrowski  and  Garstka,  1998;  Alberts, 
Garstka,  and  Stein,  1999;  Alberts  and  Hayes,  2003;  National  Research  Council,  2000),  including  a  recent  study 
on  implications  for  command  and  control  (Alberts  and  Hayes,  2006). 
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6  “Possibility  space”  spans  the  areas  not  only  of  name-level  scenarios  (e.g.,  war  with  Iran),  but  also  the  vast 
range  of  circumstances  that  apply  to  any  one  such  name-level  scenario.  This  use  of  “possibility  space”  is  equiva¬ 
lent  to  the  use  of  “scenario  space”  in  other  publications  (Davis,  2002a). 

7  The  theme  of  adaptiveness  has  been  emphasized  for  some  years  (Davis,  1994)  and  helped  motivate  the  move  to 
capabilities-based  planning  (Davis,  2002a),  but  it  is  only  now  becoming  mainstream.  Its  core  role  is  also  empha¬ 
sized  in  recent  work  by  researchers  in  Australia’s  Defense  Science  and  Technology  Organization  (Grisogono, 
2006a  and  b),  drawing  heavily  on  lessons  from  natural  complex  adaptive  systems. 

8  This  understates  the  point.  Traditional  analysis  is  often  considered  by  such  planners  as  being  irrelevant 
because  it  omits  so  many  key  factors,  including  human  action. 

9  See  Gordon  and  Trainor  (2006)  and  Franks  and  McConnell  (2004). 

10  Although  the  literature  on  real-options  theory  is  now  quite  extensive,  including  recent  books  (Mun,  2005; 
Copeland  and  Antikarov,  2003),  we  benefited  particularly  from  a  workshop  presentation  by  Scott  Matthews, 
including  his  discussion  of  actual  use  at  Boeing  in  recent  years,  based  significantly  on  a  simplified  mathematics 
that  makes  the  theory  more  practically  accessible  (Datar  and  Matthews,  2004).  Interestingly,  some  aspects  of 
Boeing’s  approach  relate  closely  to  what  we  have  referred  to  in  this  paper  as  exploratory  analysis  under  uncer¬ 
tainty,  multiresolution  modeling,  and  family  of  tools. 

11  Progress  has  been  made.  Many  studies  have  numerous  excursions,  but  the  extent  of  these  falls  far  short  of 
what  is  needed. 

12  A  series  of  RAND  reports  describes  these  methods  and  tools  (Hillestad  and  Davis,  1998;  Davis,  2002a; 
Davis,  Kulick,  and  Egner,  2005).  See  also  the  broad  discussion  of  related  issues  by  MITRE  (Kuskey,  2001). 

13  Variations  of  this  depiction  have  appeared  elsewhere  (Davis,  2002a  and  2004). 

14  The  need  for  a  family-of-tools  approach — and  the  need  in  particular  to  give  more  emphasis  to  relatively 
simple,  understandable,  but  far-reaching  analytic  methods — has  been  noted  by  others  over  time,  including  a 
Defense  Science  Board  in  1996  and,  in  2005,  a  Senior  Advisory  Group  for  M&S  chaired  by  Admiral  Dennis 
Blair  of  IDA. 

15  Exceptions  exist.  Some  simple  models  are  actually  high-resolution  specialty  models,  such  as  a  model  describ¬ 
ing  the  kinematics  of  an  individual  missile  impacting  in  an  open  area  through  which  discrete  armed  vehicles  are 
moving  (Davis,  Bigelow,  and  McEver,  2000). 

16  Current  strategic  simulations  are,  of  course,  imperfect.  JICM  lacks  even  an  aggregate-level  treatment  of  com¬ 
mand  and  control;  Thunder  has  a  weak  ground  model  and  was  intended  for  mission-level  work;  ITEM  (Inte¬ 
grated  Theater  Engagement  Model)  is  specialized  to  navy  operations  and  is  neither  agile  nor  broad.  JWARS  is  a 
much  richer  simulation  than  JICM,  but  it  is  much  more  appropriate  for  selective  in-depth  analysis  than  synoptic 
work  or  systems  analysis,  much  less  the  exploratory  analysis  discussed  in  this  paper.  JWARS  is  being  renamed 
by  U.S.  Joint  Forces  Command  as  the  Joint  Analysis  System  (JAS). 

17  A  number  of  the  relevant  models  or  modeling  systems  are  discussed  briefly  in  another  paper  (Dechant  et  ah, 
2006).  Analysis  with  such  ABMs  is  not  straightforward  (Sanchez  and  Tucas,  2000). 

18  One  example  is  analysis  of  ground  combat  with  effects  of  missiles  and  aircraft,  and  with  advanced  tactics 
such  as  the  use  of  robotics  and  precision  weapons  (Matsumura  et  ah,  2000).  Other  examples  include  analysis 
with  the  venerable  Brawler  model  of  air-to-air  engagements  and  analysis  with  the  Naval  Simulation  System 
(NSS)  (Stevens,  2000). 

19  Access  to  detailed  models  or  their  data  is  crucial  even  for  higher-level  analysis.  A  common  problem  with 
higher-level  simple  models  is  that  they  incorporate  “naive  aggregations”  that  do  violence  to  the  problem,  as  when 
they  assume  perfect  command  and  control  or  ignore  underlying  tactics,  such  as  concentration  of  force.  Better 
simple  models  incorporate  factors  reflecting  the  aggregate  effect  of  such  phenomena  and  the  price  of  hedges. 
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For  example,  a  simple  model  might  have  parameters  that  vary  depending  on  whether  one  side  or  the  other  has 
information  dominance.  How  to  aggregate  results  of  bottom- up  agent-based  modeling  is  not  yet  understood. 

20  Many  good  references  exist,  both  classic  and  modern  (Keegan,  1983;  Gordon  and  Trainor,  2006).  A  new 
book  describes  human  performance  in  military  operations,  both  historical  and  experimental  (Rowland,  2006). 

21  An  example  is  the  work  of  Dupuy  (1987). 

22  Exceptions  exist:  the  Army’s  Concepts  Analysis  Agency  has  long  funded  historical  research;  the  Joint  Staff’s 
J-8  has  long  conducted  war  games  (but  not  particularly  to  inform  analysis);  and  many  analysts  outside  of  M&S 
groups  build  or  use  small  spreadsheet-level  models  for  particular  purposes. 

23  Analysis  can  be  and  often  is  done  without  M&S.  Simulations  are  merely  a  particular  class  of  models  that 
generate  descriptions  of  system  behavior  over  time. 

24  This  point  has  been  made  repeatedly  in  review  studies  (National  Research  Council,  2004  and  2005;  Davis, 
2002a). 

25  Some  discussions  of  adaptiveness  associate  it  with  learning,  by  analogy  to  how  organisms  “learn.”  In  this 
paper,  adaptiveness  has  a  more  general  meaning  that  includes,  e.g.,  preplanned  contingency  branches,  a  rough 
thinking-through  of  how  to  respond  to  surprise  events,  and  “agility.” 

26  The  recent  agent-based  modeling  literature  is  extensive  and  includes  a  review  (Uhrmacher  and  Swartout, 
2003),  applications  to  military  problems  (Ilachinski,  2004;  Horne  and  Leonardi,  2001),  and  social-network 
problems  (Prieutula,  Carley,  and  Gasser,  1998),  among  others.  Agents  are  discussed  as  well  in  a  recent  National 
Academy  report  (National  Research  Council,  2006). 

27  For  example,  a  top-down  agent  architecture  (i.e.,  “Belief,  Desires,  Intentions”)  has  been  applied  to  a  social 
power  structure  (Taylor  et  ah,  2004)  and  a  traditional  heavy  agent  architecture  (“Soar”)  has  been  applied  in 
social  modeling  (Carley  and  Prieutala,  1993). 

28  An  agent  may,  for  example,  reason  with  a  small  number  of  high-level  factors  and  qualitative  logic,  rather 
much  as  a  human  decisionmaker  summarizes  reasoning  before  acting.  The  human  may  have  gotten  to  that  point 
by  a  far  more  tortuous  path. 

29  One  set  of  well-developed  tools  is  Repast  (North,  Collier,  and  Vos,  2006),  from  the  University  of  Chi¬ 
cago,  and  Argonne  National  Laboratory  Source  code  is  freely  available  at  http://repast.sourceforge.net  (as  of 
December  18,  2006).  We  thank  Peter  Thompson  (University  of  South  Australia)  and  Mark  Altaweel  (Argonne 
National  Laboratories)  for  information  on  Repast. 

30  Sponsored  by  the  Office  of  Naval  Research,  one  report  (Taylor  et  al.,  2002)  describes  a  generic  toolkit  to  visu¬ 
alize  and  report  on  the  internal  reasoning  process  of  an  intelligent  agent.  Also,  artificial  intelligence  researchers 
at  the  Institute  for  Creative  Technologies  report  on  a  framework  to  enable  “explainable  AI  (XAI)”  (Core  et  al., 
2006). 

31  Game-theoretic  methods  have  been  used  effectively  within  simulations — e.g.,  1980s  TAC-SAGE  work  at 
RAND  led  by  Richard  Hillestad  and  the  even  earlier  SABER  GRAND  work  in  Air  Force  Studies  and  Analysis 
led  by  Major  Leon  Goodson  (later  Brigadier  General)  under  Lieutenant  Glenn  Kent  (2002). 

32  See  the  historical  discussion  in  a  recent  biography  of  Herman  Kahn  (Ghamari-Tabrizi,  2005). 

33  These  uses  apply  also  to  social-policy  problems,  as  illustrated  by  so-called  foresight  methods  (van  de  Riet,  van 
het  Loo,  and  Kahan,  2005;  Kahan  and  Davis,  2006). 

34  A  good  deal  of  information  about  serious  games  can  be  found  at  http://en.wikipedia.org/wiki/Serious_game 
or  at  http://www.seriousgames.org  (as  of  December  18,  2006). 

35  See  discussion  in  Davis  (2001,  2002b,  and  2004). 
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36  Red  teams  apply  well  to  counterterrorism  also,  as  demonstrated  in  a  study  led  by  James  Miller  (Murray,  2002 
and  2003;  Sinnreich,  2002;  Murdock,  2003).  The  Defense  Science  Board  has  discussed  red  teaming  at  some 
length  (Defense  Science  Board,  2003). 

37  The  Army  War  College,  for  example,  has  long  developed  databases  on  what  and  how  many  forces  have  his¬ 
torically  been  used  in  various  contingencies.  A  recent  RAND  study  (Dobbins  et  al.,  2003)  reviews  historical 
evidence  on  the  perceived  need  for  forces  in  the  stabilization  and  reconstruction  phases.  Such  studies  sometimes 
fail  to  convince  skeptics  because  they  show  what  was  used,  rather  than  what  was  necessarily  required  or  what 
may  be  needed  today,  but  the  information  is  nonetheless  useful  and  often  sobering. 

38  This  is  described  in  a  short  review  paper  (Davis,  2002b). 

39  Such  operations  were  routine  with  the  RAND  Strategy  Assessment  System  (RSAS),  the  broader  predecessor 
to  the  current-day  JICM.  The  RSAS  had  libraries  of  “analytic  war  plans”  composed  of  building-block  operations 
(Schwabe,  1990).  The  RSAS  exploited  a  high-level  language  (RAND-Abel)  and  special  structures  motivated  by 
artificial-intelligence  research.  JICM  has  an  interface  language  that  provides  much  of  the  same  functionality 
(Jones,  unpublished).  The  more  detailed  JWARS,  now  being  tested  by  the  U.S.  Joint  Forces  Command  and 
others,  has  significant  capabilities  for  user-defined  rules  that  exist  in  data  and  are  easily  changed  (Burdick  et  al., 
2002).  These  are  compelling  existence  proofs  of  what  can  be  done. 

40  The  process  of  model,  game,  model  has  often  been  used  effectively  over  the  years,  dating  at  least  back  to  the 
predecessor  of  J-8  (SAGA).  For  many  years,  the  most  careful  Joint  Staff  analyses  were  done  with  TACWAR  used 
as  a  game  board  and  bookkeeper,  but  with  military  officers  making  key  decisions  along  the  way,  on  both  force 
employment  and,  sometimes,  adjudication  of  results.  The  Joint  Forces  Command  today  uses  a  model-game- 
model  approach. 

41  Modularity  in  complex  systems  was  described  well  decades  ago  by  Nobelist  Herbert  Simon  (1981).  A  number 
of  modern  books  deal  with  software  development  (Baldwin  and  Clark,  2000),  sometimes  under  the  rubric  of 
component-based  software  (Szyperski,  2002). 

42  The  state  of  the  art  in  model  composability  has  been  reviewed  recently  for  the  DoD  context,  with  many 
admonitions  for  pursuing  it  more  effectively  (Davis  and  Anderson,  2004;  National  Research  Council,  2006). 

43  Another  reason  for  “push  back”  against  suggestions  to  emphasize  modularity  is  that  the  interactions  across  a 
complex  model,  such  as  a  strategic  simulation,  are  numerous  and  important.  Overzealous  efforts  to  modularize 
can  omit  important  relationships  (as  in  limiting  air-to-ground  effects  to  direct  killing  of  vehicles).  With  older 
technology,  it  was  very  difficult  to  deal  well  with  modules  having  too  many  interactions.  Today,  that  is  much 
less  so  the  case. 

44  An  imperfect  example  is  RAND’s  START  model,  written  as  an  Excel  program.  START  was  built  as  a  spe¬ 
cialty-program  spin-off  of  JICM,  one  that  made  it  easier  to  conduct  various  analyses  for  the  U.S.  Air  Force. 
Originally,  its  predictions  were  identical  to  those  of  JICM  for  constant  data;  later,  changes  to  START  and  JICM 
in  different  projects  caused  the  models  to  diverge  somewhat.  If  there  were  reasons  to  do  so,  minor  reprogram¬ 
ming  to  reestablish  complete  consistency  would  probably  be  straightforward. 

45  The  word  “improve”  is  chosen  carefully.  There  are  practical  limits  on  the  degree  of  modularity  that  can  be 
achieved  in  the  big  models  because,  as  noted  earlier,  the  number  of  interactions  is  large. 

46  Dell  Tunceford  refers  to  such  models  as  “boutique  models”  and  has  long  championed  them  in  preference  to 
continued  exclusive  emphasis  on  large,  complex,  and  monolithic  models. 

47  This  is  based  on  a  comparison  with  the  situation  described  in  an  early-1990s  paper  (Davis  and  Blumenthal, 
1991),  which  was  quite  controversial  at  the  time.  The  conclusions  were  subsequently  reiterated  and  strengthened 
in  a  National  Academy  study  (National  Research  Council,  1997). 

48  Other  working  groups  focused  on  M&S  for  traditional  and  nontraditional  warfare,  respectively  (Bracken  et 
al.,  2006;  Dechant  et  al.,  2006). 


